The collection of the Rijksmuseum was built over a period of 200 years and did not originate from a royal collection incorporated into a national museum. Its origins were modest, with its collection fitting into five rooms at the Huis ten Bosch palace in The Hague. Although the seventeenth century was beginning to be recognized as the key period in Dutch art, the museum did not then hold paintings by Frans Hals, Rembrandt, Jan Steen, Johannes Vermeer, or Jacob van Ruisdael. The collection was built up by purchase and donation. Napoleon had carried off the stadholder's collection to Paris; the paintings were returned to The Netherlands in 1815 but housed in the Mauritshuis in The Hague rather than the Rijksmuseum. With the founding of the Rijksmuseum in 1885, holdings from other entities were brought together to establish the Rijksmuseum's major collections.

In 1795, the Batavian Republic was proclaimed; its Minister of Finance Isaac Gogel argued that a national museum, following the French example of The Louvre, would serve the national interest. On 19 November 1798, the government decided to found the museum.

On 31 May 1800, the National Art Gallery (Dutch: Nationale Kunst-Galerij), precursor of the Rijksmuseum, opened in Huis ten Bosch in The Hague. The museum exhibited around 200 paintings and historic objects from the collections of the Dutch stadtholders. In 1805, the National Art Gallery moved within The Hague to the Prince William V Gallery, on the Buitenhof. In 1806, the Kingdom of Holland was established by Napoleon Bonaparte. On the orders of king Louis Bonaparte, brother of Napoleon, the museum moved to Amsterdam in 1808. Paintings owned by that city, such as The Night Watch by Rembrandt, became part of the collection. In 1809, the museum opened in the Royal Palace in Amsterdam.

In 1817, the museum moved to the Trippenhuis. The Trippenhuis turned out to be unsuitable as a museum. In 1820, the historical objects were moved to the Mauritshuis in The Hague and in 1838, the 19th-century paintings "of living masters" were moved to King Louis Bonaparte's former summer palace Paviljoen Welgelegen in Haarlem.

In 1863, there was a design contest for a new building for the Rijksmuseum, but none of the submissions was considered to be of sufficient quality. Pierre Cuypers also participated in the contest and his submission reached the second place.

In 1876, a new contest was held and this time Pierre Cuypers won. The design was a combination of gothic and renaissance elements. The construction began on 1 October 1876. On both the inside and the outside, the building was richly decorated with references to Dutch art history. Another contest was held for these decorations. The winners were B. van Hove and Frantz Vermeylen  for the sculptures, Georg Sturm  for the tile panels and painting and W.F. Dixon for the stained glass. The museum was opened at its new location on 13 July 1885.

In 1890, a new building was added a short distance to the south-west of the Rijksmuseum. As the building was made out of fragments of demolished buildings, the building offers an overview of the history of Dutch architecture and has come to be known informally as the 'fragment building'. It is also known as the 'south wing' and is currently (in 2013) branded the Philips Wing.

In 1906, the hall for The Night Watch was rebuilt. In the interior more changes were made between the 1920s and 1950s – most multi-coloured wall decorations were painted over. In the 1960s exposition rooms and several floors were built into the two courtyards. The building had some minor renovations and restorations in 1984, 1995–1996 and 2000.

A renovation of the south wing of the museum, also known as the 'fragment building' or 'Philips Wing', was completed in 1996, the same year that the museum held its first major photography exhibition featuring its extensive collection of 19th-century photos.

In December 2003, the main building of the museum closed for a major renovation. During this renovation, about 400 objects from the collection were on display in the 'fragment building', including Rembrandt's The Night Watch and other 17th-century masterpieces.

The restoration and renovation of the Rijksmuseum are based on a design by Spanish architects Antonio Cruz and Antonio Ortiz. Many of the old interior decorations were restored and the floors in the courtyards were removed. The renovation would have initially taken five years, but was delayed and eventually took almost ten years to complete. The renovation cost €375 million.

The reconstruction of the building was completed on 16 July 2012. In March 2013, the museum's main pieces of art were moved back from the 'fragment building' (Philips Wing) to the main building. The Night Watch returned to the Night Watch Room, at the end of the Hall of Fame. On 13 April 2013, the main building was reopened by Queen Beatrix. On 1 November 2014, the Philips Wing reopened with the exhibition Modern Times: Photography in the 20th Century.

In 2025, the Rijksmuseum announced plans to partner with the municipality of Eindhoven to build a satellite branch in Eindhoven; the branch will show works from the Rijksmuseum’s collection.

On 13 January 2026, the Rijksmuseum announced plans for a new sculpture garden with three pavilions and works by Henry Moore, Alberto Giacometti, and Louise Bourgeois, among others. The new garden will be funded by the Don Quixote Foundation, as well as providing long-term loans of works for the garden. The museum hopes it will open in autumn 2026.