tsm to parquet

[TriStarGod]

Describe Development Experience Issue:

I use influx 1.8. My objective is to convert tsm to parquet as fast as possible. I'm unfamiliar with Go but learning it while exploring this project. After some searching, I found pull #25297 that led me to commit https://github.com/influxdata/influxdb/tree/dc7da40fde3b7b7f417231ca50e1196a49fb7218 . However, using the provided Dockerfile, I can't build/run the project. I tried updating some aspects like the Go version, Go packaging, make tools, and even rustup but its still failing.

Steps to reproduce:

1. git clone https://github.com/influxdata/influxdb/tree/dc7da40fde3b7b7f417231ca50e1196a49fb7218
2. navigate into project
3. run Docker / Dockerfile
4. ...

Desired result:

Build influx project.

Actual result:

Project failing to build.

Hardware Environment:

• Package: laptop / Docker / Linux Containers
• CPU: 10 vCPU
• Memory: 12 GB

Operating System:

Windows 11 Pro

Code Editing Tool:

VS Code

Build Environment:

[ ] I'm using sccache

Docker / Linux Containers

How does one build and run influx projects, preferably within Docker? @srebhan @davidby-influx

[davidby-influx]

I would strongly recommend using the master-1.x branch of the repo if you need this tooling immediately. 1.8 is a very old version. The tooling has been checked into master-1.x, and will eventually be cherry-picked to a release branch (like 1.12), but is not in one yet.

[TriStarGod]

@davidby-influx will use the master-1.x branch but how can I build it via docker?

[davidby-influx]

I do not know how your build is failing. I do not think the Dockerfile build has been maintained. Can you build via the Makefile?

[TriStarGod]

@davidby-influx

On ubuntu, I ran

• RUN DEBIAN_FRONTEND=noninteractive TZ=“America/New_York”
• apt update && apt install -y git bzr tzdata golang-go make build-essential clang pkg-clang protobuf-compiler libprotobuf-dev
• go install google.golang.org/protobuf/cmd/[email protected]
• make

It always fails with

• github.com/influxdata/flux/libflux/go/libflux; .pkg-config.sh: fork/exec ./pkg-config.sh: no such file or directory
• make: *** [Makefile:4: build] Error 1

[philjb]

For building influxd_tools that includes #25297, use this command. It is a variation of the go run instruction that is provided in that pr

cd ./influxdb
go build -o influx_tools -ldflags "-X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=ignore" ./cmd/influx_tools/

I built it like so this week.

[TriStarGod]

@philjb Thanks for commenting. Tried your command within the latest master-1x and it failed with

Package flux not found in the pkg-config search path.
Perhaps you should add the directory containing ‘flux.pc’ to the PKG_CONFIG_PATH environment variable 
Package ‘flux’, required by ‘virtual:world’, not found

I’m running all this inside the latest ubuntu docker container.

[dantony-delphinus]

Same issue. Can't build influx master-1.x within docker container.

[philjb]

@TriStarGod - ah - I must have some pkg-config configuration setup already. Try following the build steps in https://github.com/influxdata/influxdb/blob/master-1.x/CONTRIBUTING.md

@dantony-delphinus - this ticket is about building only the influx_tools binary, which are planning to include in a influxdb 1.12 release this summer (hopefully this month). The same contributing guide may help you with building influxdb from source.

Can you tell me why you are building from source instead of using the prebuilt binaries? That may help me troubleshoot better for you.

[philjb]

The homebrew formula may be helpful too: https://github.com/Homebrew/homebrew-core/blob/main/Formula/i/influxdb%401.rb

The build depends on rust and the influxd pkg-config wrapper too.

[TriStarGod]

@philjb Been trying my best to follow it. In my ubuntu container, installed gvm and go1.23.8, mkdir $HOME/gocodez, and export GOPATH=$HOME/gocodez

When I ran go get github.com/influxdata/influxdb it fails with go: go.mod file not found in current directory or any parent directory. I don't know how the old go get used to work so, I skipped this step and tried following the Cloning a fork step. I created a directory $GOPATH/src/github.com/influxdata (essentially /root/gocodez/src/github.com/influxdata). Then I cd into influxdata and git clone --single-branch --branch master-1.x https://github.com/influxdata/influxdb.git. I cd into influxdb, set the export PKG_CONFIG= ... and ran go clean ./ ... command successfully. I tried running make but it failed. I tried your go build command but it also failed. Installed rustup / cargo and now it builds and installs.

However, how do I call /run it?

[TriStarGod]

Running tools from $HOME/gocodez/bin . I do have a weird conflicted versions error if I run influx_tools directly but building via the commands outlined in pull #25297 overcome it.

[philjb]

@TriStarGod - sounds like you were able to build it!

Were you able to create parquet files with the tool?

[TriStarGod]

@philjb was working. For reliability reasons, wrote a script to set all the variables and dirs. It still builds but influx_tools fails with Flight.proto conflicted versions

[TriStarGod]

@philjb got it working. The tool builds and runs. It’s crashing after a few conversions but I’m debugging it (and learning go lol)

[philjb]

Hi @TriStarGod - What are the crash/panic error messages you are seeing? If you find anything that needs fixing or improvement, please open a pr for the changes or you can let me know what needs to happen.