td-agentをdockerで動かす

構成

td-agent/
├── docker-compose.yml
├── Dockerfile
├── test.log
├── conf
│   ├── app.conf
│   └── td-agent.conf
└── install-redhat-td-agent2.5.sh

設定

docker

FROM drecom/centos-ruby:2.6.5

ADD install-redhat-td-agent2.5.sh /tmp/

RUN sh /tmp/install-redhat-td-agent2.5.sh

RUN td-agent-gem install \
    fluent-plugin-multiprocess \
    fluent-plugin-forest \
    fluent-plugin-rewrite-tag-filter

RUN /etc/init.d/td-agent start

CMD ["td-agent"]

version: “3” services: minio: image: minio/minio:RELEASE.2020-01-25T02-50-51Z volumes: - minio_volume:/data ports: - “9090:9000” environment: MINIO_ACCESS_KEY: minio_access_key MINIO_SECRET_KEY: minio_secret_key command: server /data miniomc: image: minio/mc:RELEASE.2020-01-25T03-02-19Z depends_on: - minio entrypoint: > /bin/sh -c “ until (/usr/bin/mc config host add minio http://minio:9000 minio_access_key minio_secret_key) do echo ‘…waiting…’ && sleep 1; done; /usr/bin/mc mb minio/platform-img-develop –region=ap-northeast-1; /usr/bin/mc policy set public minio/platform-img-develop; /usr/bin/mc mb minio/app-log –region=ap-northeast-1; /usr/bin/mc policy set public minio/app-log; exit 0; “ td-agent: build: context: ./td-agent dockerfile: Dockerfile.dev volumes: - ./td-agent/conf:/etc/td-agent/ - ./test.log:/var/log/rails ports: - “24224:24224”

## td-agent
```:td-agent.conf
<source>
  type multiprocess
  <process>
    cmdline -c /etc/td-agent/app.conf --log /var/log/td-agent/td-agent.app.log
    sleep_before_start 1s
    sleep_before_shutdown 5s
    pid_file /var/run/td-agent/app.pid
  </process>
</source>

<source>
  @type tail
  format none
  path /var/log/rails/development.log
  pos_file /var/log/td-agent/rails.log.pos
  tag log.api
</source>

<match log.**>
  type forest
  subtype s3
  <template>
    output_tag false
    output_time false

    aws_key_id minio_access_key
    aws_sec_key minio_secret_key

    path ${tag_parts[1]}/

    s3_bucket app-log
    s3_endpoint http://minio:9000/
    s3_region ap-northeast-1
    s3_object_key_format %{path}%{time_slice}_%{index}-%{hostname}.%{file_extension}
    force_path_style true   # This prevents AWS SDK from breaking endpoint URL

    buffer_path /var/log/td-agent/buffer/${hostname}.${tag_parts[1]}
    time_slice_format %Y/%m/%d/%Y%m%d-%H

    buffer_chunk_limit 128m
    buffer_queue_limit 5
    flush_interval 1800s
    flush_at_shutdown true
  </template>
</match>

<match fluent.info.**>
  @type stdout
</match>

<match **>
  @type copy

  <store>
    @type file
    path /var/log/td-agent/debug
  </store>

  <store>
    @type stdout
  </store>
</match>

その他

install-redhat-td-agent2.5.sh は $ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.5.sh | sh からsudoに関する設定を抜いたもの。