fileparser

fileparser

fileparser is a Go library for parsing various tabular data formats. It provides a unified interface for reading CSV, TSV, LTSV, Parquet, and XLSX files, with optional compression support.

This package is designed to be used by filesql, fileprep, and fileframe.

• fileprep: struct-tag preprocessing and validation for CSV/TSV/LTSV, Parquet, Excel.
• filesql: sql driver for CSV, TSV, LTSV, Parquet, Excel with compression support.
• fileframe: DataFrame API for CSV/TSV/LTSV, Parquet, Excel.

Features

• Multiple formats: CSV, TSV, LTSV, Parquet, XLSX
• Compression support: gzip, bzip2, xz, zstd, zlib, snappy, s2, lz4
• Type inference: Automatically detects column types (TEXT, INTEGER, REAL, DATETIME)
• File type detection: Detects file format from path extension
• Pure Go: No CGO required for any compression format

Installation

go get github.com/nao1215/fileparser

Usage

Parsing CSV

csvData := `name,age,score
Alice,30,85.5
Bob,25,92.0
Charlie,35,78.5`

result, err := fileparser.Parse(strings.NewReader(csvData), fileparser.CSV)
if err != nil {
    log.Fatal(err)
}

fmt.Println("Headers:", result.Headers)
fmt.Println("Records:", len(result.Records))
fmt.Println("First row:", result.Records[0])

Output:

Headers: [name age score]
Records: 3
First row: [Alice 30 85.5]

Parsing TSV

tsvData := `id	product	price
1	Laptop	999.99
2	Mouse	29.99
3	Keyboard	79.99`

result, err := fileparser.Parse(strings.NewReader(tsvData), fileparser.TSV)
if err != nil {
    log.Fatal(err)
}

fmt.Println("Headers:", result.Headers)
fmt.Println("Records:", len(result.Records))

Output:

Headers: [id product price]
Records: 3

Parsing LTSV

ltsvData := `host:192.168.1.1	method:GET	path:/index.html
host:192.168.1.2	method:POST	path:/api/users`

result, err := fileparser.Parse(strings.NewReader(ltsvData), fileparser.LTSV)
if err != nil {
    log.Fatal(err)
}

fmt.Println("Headers:", result.Headers)
fmt.Println("First row:", result.Records[0])

Output:

Headers: [host method path]
First row: [192.168.1.1 GET /index.html]

Auto-detect File Type

paths := []string{
    "data.csv",
    "data.csv.gz",
    "report.xlsx",
    "logs.ltsv.zst",
    "analytics.parquet",
}

for _, path := range paths {
    ft := fileparser.DetectFileType(path)
    fmt.Printf("%s -> %s\n", path, ft)
}

Output:

data.csv -> CSV
data.csv.gz -> CSV (gzip)
report.xlsx -> XLSX
logs.ltsv.zst -> LTSV (zstd)
analytics.parquet -> Parquet

Check Compression

types := []fileparser.FileType{
    fileparser.CSV,
    fileparser.CSVGZ,
    fileparser.Parquet,
    fileparser.ParquetZSTD,
}

for _, ft := range types {
    fmt.Printf("%s compressed: %v\n", ft, fileparser.IsCompressed(ft))
}

Output:

CSV compressed: false
CSV (gzip) compressed: true
Parquet compressed: false
Parquet (zstd) compressed: true

Get Base File Type

types := []fileparser.FileType{
    fileparser.CSV,
    fileparser.CSVGZ,
    fileparser.TSVBZ2,
    fileparser.ParquetZSTD,
}

for _, ft := range types {
    base := fileparser.BaseFileType(ft)
    fmt.Printf("%s -> %s\n", ft, base)
}

Output:

CSV -> CSV
CSV (gzip) -> CSV
TSV (bzip2) -> TSV
Parquet (zstd) -> Parquet

Convert String Values to Typed Values

// Integer column
intVal := fileparser.ParseValue("42", fileparser.TypeInteger)
fmt.Printf("Integer: %v (%T)\n", intVal, intVal)

// Real column
realVal := fileparser.ParseValue("3.14", fileparser.TypeReal)
fmt.Printf("Real: %v (%T)\n", realVal, realVal)

// Text column
textVal := fileparser.ParseValue("hello", fileparser.TypeText)
fmt.Printf("Text: %v (%T)\n", textVal, textVal)

// Empty value returns nil
nilVal := fileparser.ParseValue("", fileparser.TypeInteger)
fmt.Printf("Empty: %v\n", nilVal)

Output:

Integer: 42 (int64)
Real: 3.14 (float64)
Text: hello (string)
Empty: <nil>

Automatic Column Type Inference

csvData := `id,name,score,date
1,Alice,85.5,2024-01-15
2,Bob,92.0,2024-01-16
3,Charlie,78.5,2024-01-17`

result, err := fileparser.Parse(strings.NewReader(csvData), fileparser.CSV)
if err != nil {
    log.Fatal(err)
}

for i, header := range result.Headers {
    fmt.Printf("%s: %s\n", header, result.ColumnTypes[i])
}

Output:

id: INTEGER
name: TEXT
score: REAL
date: DATETIME

Supported File Types

ACH (NACHA) Support - Experimental

Warning: ACH file support is experimental. The API may change or delete in future versions.

The ach subpackage provides support for parsing ACH (Automated Clearing House) files following the NACHA format. ACH files are converted to multiple TableData structures for SQL querying.

Supported Tables

Limitations

Read-only fields: The following fields are exported for viewing but changes are not written back to ACH files:

• IAT Addenda sequence numbers (entry_detail_sequence_number, sequence_number)

Addenda05 index behavior: When an entry has multiple addenda types (e.g., Addenda02 + Addenda05), the addenda_index represents the position within all addenda for that entry, not the index within Addenda05 specifically. For updates, use addenda_type = '05' to filter correctly.

Validation: Modifying ACH data via SQL may create invalid ACH files. The moov-io/ach library's Create() method will validate the file, but users should ensure data consistency (e.g., AddendaRecordIndicator matches actual addenda presence).

Usage

import "github.com/nao1215/fileparser/ach"

// Parse ACH file
file, err := os.Open("payment.ach")
if err != nil {
    log.Fatal(err)
}
defer file.Close()

tableSet, err := ach.Parse(file)
if err != nil {
    log.Fatal(err)
}

// Access tables
entries := tableSet.GetEntriesTable()
fmt.Printf("Found %d entries\n", len(entries.Records))

Compression Formats

Column Types

The parser automatically infers column types based on the data:

License

MIT License. See LICENSE for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.