“The document is intended only for interested parties; please do not show it on Google.” This sentence does not yet decide who may actually open the file. Perhaps a general viewing checklist should remain reachable without search hits. Perhaps, however, the file contains personal information and may be sent only to expressly authorised persons. The technical task is different in both cases.
This workshop is aimed at real-estate companies with their own website or contracted website maintenance in Austria and Germany. It is a standalone procedure for clear commissioning. Neither a third-party website nor a protected area is examined or altered here. Concrete protective measures must be professionally determined and implemented for the specific environment.
Three questions instead of an imprecise word
First describe the target group: May any person with the address read the file, or is actual authorisation required? Then ask about the search representation: Should the document be findable via search engines? Only afterwards comes the question which automated fetches are desired and how the technical maintenance handles them.
This sequence prevents a typical fallacy. “Not findable” is not a precise description of “confidential”. A file deliberately omitted from a navigation can also have its own address that can be called. Whether there is an effective protection there is therefore a separate review task.
In our fictional office three documents are present: a general moving checklist, a text preview not yet released and a case-related document. The checklist can be intended as a public information aid. The preview should initially be accessible only to the agreed editorial team. For the case document the required recipients, scope of data and suitable transmission method are clarified in advance. An identical upload folder alone would not be proof that these three objectives are met.
What robots.txt actually describes
Google explains robots.txt as a means to control crawling traffic. The documentation explicitly warns against relying on it to hide text-based websites and PDFs from search reliably. A blocked URL can under certain circumstances still appear in search results. In addition, not all crawlers follow such instructions. [ROBOTS]
For your commission this means: A statement like “the folder is listed in robots.txt” does not answer the question whether unauthorised persons can open the file. Demand a separate description of the intended access. This article does not recommend a specific server configuration and does not explain any single setting as part of a complete security concept.
What noindex is used for
Google describes noindex as a rule to prevent inclusion in the supporting search index. For Google to recognise it, the crawler must be able to fetch the resource in question; a simultaneous crawling ban can precisely prevent this recognition. For content not presented as HTML the documentation names, among other things, an appropriate HTTP response header. [NOINDEX]
This is relevant for publicly readable content with a desired restricted search representation. It still does not answer which person may read a confidential file. The intention “out of the Google index” and the intention “only for authorised recipients” therefore belong as different requirements in the commission.
Have technical details transferred to the actual website by the responsible maintenance. A code fragment from an advisory article would, without knowledge of delivery and the other rules, not constitute a reliable acceptance. This text likewise promises neither a specific search result nor an immediate removal.
Consider the file and its environment together
The visible download button is only part of the distribution path. In your own commission the actual file address, previews if applicable and deliberately offered alternative versions should be named. Otherwise the overview page may be restricted while another version remains available by other means.
In the example the team removes the preview button from a public page. The task is then initially considered complete internally. On follow-up it turns out that only the button was checked, not the agreed access restriction of the file. The team therefore corrects the review task. It does not assume either an actual data leak or an automatically secure new solution.
A later authorised check should name the specific file version and the intended user state. “Opened as the responsible editor” does not address the same issue as “not accessible as an unauthorised person”. How such checks are carried out safely and lawfully is determined by the responsible specialist.
An assignment that both sides understand the same way
Write a short sentence for each document about purpose, recipient group and desired search presentation. Add the specific version and the person responsible. For confidential documents an appropriate protection will be commissioned separately; for public downloads the publication of the cleaned version will be reviewed.
The acceptance then describes exactly what was found: for example the visibility of the released checklist or the professionally reviewed access restriction of a designated preview. Open points such as older distribution channels remain documented as open. The mere exchange of a link must not make this issue invisible.
Already distributed copies are also a separate working status. A changed web link is not proof that every previously stored file has disappeared. If there is suspicion of an unauthorised disclosure, that specific case should be reported promptly to the responsible specialists and entered into the designated incident process; this guide does not assess any reporting obligations.
The result is a comprehensible distribution decision with an appropriate technical assignment. You do not have to write every server rule yourself for this. However, you should be able to understand whether the discussion concerns search presentation, automated retrievals or actual reading permissions.