Hungary has declared a state of emergency in two southern counties as tough new laws to stop migrants entering illegally came into force.
Police said they had arrested 60 people accused of trying to breach a razor-wire fence on the border with Serbia.
The state of emergency gives police extra powers and would allow troop deployments if parliament approves.
The EU is facing a huge influx of migrants, many fleeing conflict and poverty in countries including Syria.
Meanwhile, Germany and Austria are calling for a special meeting of EU leaders next week to discuss the crisis.
German Chancellor Angela Merkel told a news conference that “this problem can only be solved together. It is a responsibility for the entire European Union”.
The EU’s border agency said more than 500,000 migrants had arrived at the EU’s borders this year, compared with 280,000 in 2014. The vast majority have come by boat across the Mediterranean.
A boat following the most popular recent route, between Turkey and Greece, sank on Tuesday leaving 22 people dead, Turkish media reported.
Starting on Tuesday, the EU has agreed to relocate 40,000 migrants from Greece and Italy to other EU states. But it has yet to agree on mandatory quotas for a further 120,000 asylum seekers.
After the new Hungarian laws came into effect at midnight (22:00 GMT Monday), police sealed a railway crossing point that had been used by tens of thousands of migrants.
Around midday there were tense scenes as hundreds streamed towards the fence, some searching for a way through and others starting a sit-down strike, throwing down food and water in protest at not being granted passage.
Police buses will now take asylum applicants to registration centres, but if their applications are refused they will now be returned to Serbia rather than being given passage through Hungary, the BBC’s Nick Thorpe reports from the border.
Hungarian authorities said more than 9,000 – a new record – crossed into the country before the border was closed on Monday. Some 20,000 crossed into Austria.
From Tuesday, anyone who crosses the border illegally will face criminal charges, and 30 judges have been put on standby to try offenders.
The laws also make it a criminal offence – punishable by prison or deportation – to damage the newly-built four-metre (13ft) fence along Hungary’s 175km (110 mile) border with Serbia.
Hungarian government spokesman Zoltan Kovacs said “the official and legal ways to come to Hungary and therefore to the European Union remain open. That’s all we ask from all migrants – that they should comply with international and European law.”
But the Serbian minister responsible for the migrant crisis, Aleksandar Vulin, argued the closure was unsustainable. “They will have to open the border,” he said.
The European Commission said it was seeking clarification of parts of the new legislation, to check whether it is in line with EU asylum rules.
At talks in Brussels on Monday, a majority of states had agreed in principle to the idea of relocating a further 120,000 through mandatory quotas, and there was hope the proposal could be finally approved at a meeting on 8 October.
However, the Czech Republic, Slovakia and Hungary have opposed the quotas.
“There was no consensus, several countries disagreed,” Slovak Interior Minister Robert Kalinak said after the talks.
German Interior Minister Thomas de Maiziere said on Tuesday there should be ways of “exerting pressure” on states that refused binding quotas, possibly by reducing the amount of EU funding they receive.
But the Czech state secretary for the EU, Toma Prouza, said such threats were “empty but very damaging to all”
A spokeswoman from UN refugee agency, Melissa Fleming, said she expected migrant “chaos” to continue in the absence of more decisive action by the EU, with migrants seeking a new route.
Germany introduced temporary border controls on Monday. That slowed down the passage of migrants from Austria, where about 2,000 people slept in railway stations overnight.
Austria – one of several EU countries to say it would tighten border controls – is starting to deploy hundreds of troops to help the police deal with migrant arrivals.
The moves are a challenge to the EU’s Schengen agreement on free movement, although the rules do allow for temporary controls in emergencies.