The museum building was originally a railway station, Gare d'Orsay, located next to the Seine river. Built on the site of the Palais d'Orsay, its central location was convenient for commuting travelers. The station was constructed for the Chemin de Fer de Paris à Orléans and finished in time for the 1900 Exposition Universelle to the design of three architects: Lucien Magne, Émile Bénard and Victor Laloux. The Gare d'Orsay design was considered to be an "anachronism". The museum building is accountable to 19th-century aesthetics and construction techniques that were molded to accommodate modern life. Paris had become a modern capital city following grand changes implemented by Napoleon III and Georges-Eugène Haussmann. So architecture was allocated a permanent exhibition area in the museum showcasing the new Paris opera house, the Palais Garnier designed by Charles Garnier and built from 1863 to 1875. At the end of the museum's central aisle a space was dedicated to town planning, architecture and decoration.

In the 1970s work began on building a 1 km-long tunnel under the station as part of the creation of line C of the Réseau Express Régional with a new station under the old station. In 1970, permission was granted to demolish the station but Jacques Duhamel, Minister for Cultural Affairs, ruled against plans to build a new hotel in its stead. The station was put on the supplementary list of Historic Monuments and finally listed in 1978. The suggestion to turn the station into a museum came from the Directorate of the Museum of France. The idea was to build a museum that would bridge the gap between the Louvre and the National Museum of Modern Art at the Georges Pompidou Centre. The plan was accepted by Georges Pompidou and a study was commissioned in 1974. In 1978, a competition was organized to design the new museum. ACT Architecture, a team of three young architects (Pierre Colboc, Renaud Bardon and Jean-Paul Philippon), were awarded the contract which involved creating 20,000 square metres (220,000 sq ft) of new floorspace on four floors. The construction work was carried out by Bouygues. In 1981, the Italian architect Gae Aulenti was chosen to design the interior including the internal arrangement, decoration, furniture and fittings of the museum. The arrangement of the galleries she designed was elaborate and inhabited the three main levels that are under the museum's barrel vault atrium. On the main level of the building, a central nave was formed by the surrounding stone structures that were previously the building's train platforms. The central nave's structures break up the immense sculpture and gallery spaces and provided more organized units for viewing the art.

In July 1986, the museum was ready to receive its exhibits. It took 6 months to install the 2,000 or so paintings, 600 sculptures and other works. The museum officially opened in December 1986 by then-president François Mitterrand. At any time about 3,000 art pieces are on display within Musée d'Orsay. Within the museum is a 1:100 scale model created by Richard Peduzzi of an aerial view of Paris Opera and surrounding area. This model is encapsulated underneath glass flooring that viewers walk on as they proceed through the museum. This installation allows the viewers to understand the city planning of Paris at the time, which has made this attraction one of the most popular within the museum.

Another exhibit within the museum is "A Passion for France: The Marlene and Spencer Hays Collection". This collection was donated by Marlene and Spencer Hays, art collectors who reside in Texas and have been collecting art since the early 1970s. In 2016 the museum complied to keeping the collection of about 600 art pieces in one collection rather than dispersed throughout other exhibits. Since World War II, France has not been donated a collection of foreign art this large. The collection favors mostly post-impressionist works. Artists featured in this collection are Bonnard, Vuillard, Maurice Denis, Odilon Redon, Aristide Maillol, André Derain, Edgar Degas, and Jean-Baptiste-Camille Corot. To make room for the art that has been donated, the Musée d'Orsay is scheduled to undergo a radical transformation over the next decade, 2020 on. This remodel is funded in part by an anonymous US patron who donated €20 million to a building project known as Orsay Grand Ouvert (Orsay Wide Open). The gift was made via the American Friends of the Musées d'Orsay et de l'Orangerie. The projected completion date is 2026, implementing new galleries and education opportunities to endorse a conductive experience.

The square next to the museum displays six bronze allegorical sculptural groups in a row, originally produced for the Exposition Universelle: