astro-lib icon indicating copy to clipboard operation
astro-lib copied to clipboard

Astro robots not generating a robots file or a meta tag

Open gplusdotgr opened this issue 2 years ago • 0 comments

Here's my astro config:

import { defineConfig } from "astro/config"
import node from "@astrojs/node"
import vue from "@astrojs/vue"
import svgLoader from "@gkatsanos/vite-svg-loader"
import tailwind from "@astrojs/tailwind"
import robotsTxt from "astro-robots-txt"

// https://astro.build/config
export default defineConfig({
  server: {
    port: 3000,
    host: "0.0.0.0",
  },
  output: "server",
  adapter: node({
    mode: "standalone",
  }),
  build: {
    assets: "requests-frontend/_astro",
  },
  vite: {
    server: {
      proxy: {
        "^/octopus/api/.*": {
          target: "https://www.wlw-staging.de",
          changeOrigin: true,
        },
      },
    },
    plugins: [
      svgLoader({
        defaultImport: "url", // or 'raw'
      }),
    ],

    ssr: {
      noExternal: ["path-to-regexp"],
    },
    optimizeDeps: {
      // Why? dd-trace attempts to patch graphql, but it's not a dependency of this project.
      exclude: ["graphql"],
    },
  },
  integrations: [
    vue(),
    tailwind(),
    robotsTxt({
      policy: [
        {
          userAgent: "*",
          // The next line enables or disables the crawling on the `robots.txt` level
          disallow: "/",
        },
      ],
    }),
  ],
})

Looking at the HTML and the network tab, dont see a robots file or a meta tag.. whats up? btw my Astro app is served from a sub-directory ( www.example.com/myastroapp/ ) with an nginx webserver on top of it.

gplusdotgr avatar Jun 15 '23 08:06 gplusdotgr