Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: dataform-co/dataform
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 3.1.3
Choose a base ref
...
head repository: dataform-co/dataform
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 3.1.4
Choose a head ref
  • 6 commits
  • 58 files changed
  • 4 contributors

Commits on Nov 3, 2026

  1. feat(cli): create schemas lazily (#2408)

    * Create schemas lazily if location unset
    
    * Wrap all tasks in an action
    
    * Remove old path; always create schemas lazily
    
    * Add lock on schema creation
    
    * Dry-run select statements in wrapped scripts
    
    * Include pre- and postops in dry run validation query
    
    * Do not wait for dataset creation if task skipped
    
    * Fixes
    
    * Split off schema creation tests into a seperate file
    
    * Split location tests to a seperate file
    
    * Adjust error position
    
    * Reformat
    
    * Only create operations schemas if hasOutput
    
    * Rebase fix
    
    * Mark actions as cancelled when cancel on waiting for schema creation
    
    * Exclude post-ops from dry run validation query
    
    * Handle raw strings in codeEndIndex
    
    * Misc fixes
    
    * Format
    
    * run.ts refactor
    
    * Restore and update proto comment
    
    * Handle line prefixes for error parsing
    
    * Minor formatting issues
    Ford544 authored Nov 3, 2026
    Configuration menu
    Copy the full SHA
    7b73674 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5ffcfa4 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2026

  1. Filter stateless-install project copies with ignore-file overrides (#…

    …2272)
    
    * Filter the stateless-install project copy using .gitignore
    
    Fixes #2269. When dataformCoreVersion is set in workflow_settings.yaml,
    compile() copies the whole project directory to a temporary directory
    before running `npm i` there. That copy had no filter, so its cost
    scaled with everything under the project root -- most commonly a large
    .venv or build-output directory sitting next to definitions/ -- rather
    than with the Dataform project itself.
    
    Adds buildProjectCopyFilter(), an fs-extra copySync filter built from
    the project's own root .gitignore (already written by `dataform init`,
    but not previously consulted anywhere in the compile pipeline), plus a
    small always-ignored floor of .git and node_modules that a .gitignore
    negation pattern cannot override.
    
    Using the project's .gitignore rather than a hardcoded list of directory
    names avoids having to enumerate every ecosystem's junk directories
    (.venv, target/, __pycache__/, vendor/, coverage/, ...), since a
    project's .gitignore already states what that project treats as
    disposable.
    
    Behavior change worth calling out in review: a gitignored file is no
    longer copied, so it is also no longer compiled. A project that
    generates definitions into a gitignored path now has to unignore that
    path. Verbose mode prints which exclusions are applied. Projects with
    no .gitignore are unaffected beyond the .git/node_modules floor.
    
    No lockfile regeneration is needed: `ignore` is already resolved in
    yarn.lock at 5.3.2 as an existing transitive dependency, and the
    declared ^5.2.0 range matches that entry.
    
    * Address review: .dataformignore support and accurate verbose log
    
    - Read an optional .dataformignore from the project root after the
      .gitignore, into the same ignore instance. It can exclude further
      paths, or un-ignore gitignored ones with `!pattern` (for example,
      definitions generated into a gitignored directory). The .git and
      node_modules floor still cannot be overridden by either file.
    - The verbose log now names the ignore files actually found, and no
      longer mentions a .gitignore when the project has none.
    - Reformat the new files with the repo's Prettier 3 config.
    
    * Match ignore patterns case-sensitively; document negation limits
    
    - ignore() matches case-insensitively by default, so on a
      case-sensitive filesystem a `definitions/staging/` pattern also
      dropped `definitions/Staging/table.sqlx`, which git keeps. Always
      match case-sensitively: at worst this copies a little extra on
      case-insensitive filesystems, and never drops a definition.
    - Document that, as in git, a file can't be re-included while an
      ancestor directory is still excluded, with a concrete example, and
      test the child-only negation case.
    
    * Harden the copy filter's fixed rules and ignore-file loading
    
    - Compare the always-ignored .git/node_modules names case-insensitively,
      so .GIT or NODE_MODULES on a case-insensitive filesystem is excluded.
    - Always copy the root workflow_settings.yaml. compile() has already
      read it from the original project, and compilation can't proceed
      without it, so a broad pattern like `*.yaml` must not drop it.
    - Skip a .gitignore/.dataformignore that isn't a regular file, instead
      of failing the compile with EISDIR.
    - Document that patterns are evaluated without consulting git's index,
      so force-added tracked files matching a pattern are still excluded.
    
    * Follow the filesystem's case sensitivity in the copy filter
    
    A fixed case policy was wrong in both directions: always-sensitive
    matching let a mixed-case negation miss its file on a case-insensitive
    filesystem, and always-folding the .git/node_modules floor excluded a
    legitimate NODE_MODULES directory on Linux.
    
    Detect once whether the project directory is case-insensitive (probing
    an existing entry under its case-swapped name, falling back to the
    platform default), and apply that to the ignore matcher, the
    always-ignored floor and the workflow_settings.yaml allowlist, as git
    does with core.ignorecase. Tests pin each mode explicitly so they
    behave the same on every platform.
    
    * Always copy compilation inputs; drop .dataformignore and case probing
    
    Address review on #2272:
    - Don't apply .gitignore to what compilation reads: definitions/,
      includes/, workflow_settings.yaml (or dataform.json) and .npmrc, nor
      to anything reachable from them through symlinks within the project,
      matching the compile glob, which follows symlinks. Only .git is
      skipped inside those paths. Everything else is still filtered, so the
      slow copy from #2269 stays fixed.
    - Remove .dataformignore support; with compilation inputs always
      copied, its main use case is gone.
    - Replace case-sensitivity probing with the platform default:
      case-insensitive on win32 and darwin, sensitive elsewhere.
    - Update the verbose log message, and document the copy's behavior in
      the dataformCoreVersion docs.
    
    * Note that caught require() failures can change behavior silently
    
    The dataformCoreVersion docs said a skipped file loaded by project code
    makes compilation fail. Code that catches the error carries on instead,
    possibly with different behavior, so say that.
    
    * Document keeping loaded gitignored files under includes/
    
    A gitignored file that project code loads isn't copied for a
    stateless-install compile. A subdirectory of includes/ is always copied
    and isn't loaded as globals, so point users there.
    
    * Qualify copy filter claims about git and relocated helpers
    
    Nested .gitignore negations can un-ignore files this filter still
    excludes, so the copy can differ from git in either direction, not just
    copy more. A caught require() failure changes behavior rather than
    failing. Moving a helper under includes/ also means moving what it loads
    by relative path.
    
    * Restore .dataformignore and explain modules the copy excluded
    
    Some files the stateless-install copy needs aren't compilation inputs:
    SQL that actions.yaml references outside definitions/, or a CA file
    .npmrc names by a relative path. When gitignored, they were dropped with
    no way to keep them short of changing .gitignore or moving them.
    
    - Read an optional root .dataformignore after .gitignore, so it can
      re-include gitignored paths or exclude more. Compilation inputs and
      the .git/node_modules floor still can't be overridden.
    - When compilation can't find a module that exists in the project but
      not in the copy, say which path an ignore file excludes and how to
      re-include it, instead of the CLI's 'Could not find NPM dependencies'
      message.
    - Document both on dataformCoreVersion, and test re-inclusion end to
      end with an actions.yaml reference.
    
    * Simplify the copy filter's error hint and tests
    
    - Find the excluded path from the finished copy, which never contains a
      directory the filter excluded, instead of rebuilding the filter. Check
      the .git/node_modules floor with the platform's case sensitivity, as
      the filter does.
    - Share the stateless-install setup across the end-to-end tests, and
      drop imports left over from a removed test.
    - Copy through copyProjectForStatelessInstall in the unit tests, drop a
      test the others cover, and fold the directory-.gitignore case into
      the findProjectIgnoreFiles test.
    - Trim the dataformCoreVersion docs.
    
    * Test the graph-error hint and a !* .dataformignore
    
    - End to end: a definition that require()s a gitignored helper gets the
      .dataformignore hint in both --json output and the console's printed
      stack, and compiles once the helper is re-included.
    - A .dataformignore of !* re-includes everything the .gitignore
      excludes, including nested patterns, apart from .git and
      node_modules.
    
    * Make the excluded-module hint descriptive rather than prescriptive
    
    The hint suggested one negation to add, which doesn't always recover the
    file: another pattern can also match it (queries/ and *.sql), and a file
    name can contain pattern syntax (example[1].sql). It now names the path
    missing from the copy and points to .dataformignore without proposing a
    pattern.
    
    - Check presence with lstat, so a copied symlink whose target was
      excluded isn't blamed on the link.
    - Start from the shallowest path missing from the copy, so an
      extensionless require("lib/helper") is explained when a directory on
      its path was excluded.
    - Say "if the module is there", since an excluded directory doesn't
      prove the module was in it.
    - Docs: call the always-copied paths what they are, not compilation
      inputs, and don't describe .gitignore as listing what's disposable.
    
    * Don't explain modules behind copied symlinks; make symlink tests real
    
    - Stop explaining a missing module once its path reaches a copied
      symbolic link. What's missing is behind the link, so naming a path
      under it (lib/helper.js for lib -> generated) sent users to re-include
      the wrong thing, and with lib -> .git/lib it blamed ignore files for
      the fixed .git exclusion.
    - Under Bazel, rules_nodejs patches fs.readlinkSync to return absolute
      paths, so links copied in-process pointed back into the original
      project. Two symlink tests read through those links and passed even
      with link following disabled. Check each link's target at its own path
      in the copy instead, and point the hint tests' copied links at their
      excluded targets in the copy.
    - Docs: note the .git exception inside definitions/ and includes/.
    
    * Print the excluded-module note separately; scan only for symlinks
    
    - Leave missing-module errors unchanged and print the note about an
      excluded path to stderr alongside them, instead of rewriting the
      error's message and stack. Guard the catch with `instanceof Error`.
    - Copy anything under the compilation input roots by path prefix. The
      pre-pass now only lists their directories to find symlinks, recording
      paths just for what links reach outside them.
    - Document and test that a directory-only pattern doesn't match a
      symlink to a directory, as in Git.
    
    * Move stateless-install docs to their own page; clean up the temp copy on failure
    
    - Shorten the dataformCoreVersion field docs to a summary that links to a new
      docs/stateless-install.md page.
    - Remove the temporary copy in a finally block, so it's removed when npm i or
      compilation fails too, and create it only on the stateless path.
    - Pass the same case setting to explainExcludedModule as to the copy filter.
    
    * Adapt to the node:vm runner's missing-module message
    
    The runner names the requiring file after the module, and no longer throws a VMError, so the CLI doesn't report it as a missing npm dependency.
    
    * Depend on ignore 7.x
    
    7.0.8 is already in the lockfile, through typescript-eslint.
    
    * Don't name the Bazel rules that patch readlinkSync in tests
    
    * Correct comments on relative module names and best-effort cleanup
    
    * fix(cli): keep temporary environment cleanup best effort
    
    A failure to remove the temporary directory mustn't turn a successful
    compile into a failure, replace the error that ended the compile, or
    throw from the process exit listener.
    dmargol1 authored Nov 5, 2026
    Configuration menu
    Copy the full SHA
    2576779 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2026

  1. Adapt PropertyGraph graph.yaml schema and DDL compiler to Semantic On…

    …tology V1 (#2417)
    
    * Adapt PropertyGraph graph.yaml schema and DDL compiler to Semantic Ontology V1
    
    * Assert INFORMATION_SCHEMA.PROPERTY_GRAPHS DDL without Enterprise reservation
    
    * Pass property graph field and metric expressions as-is into BigQuery DDL
    
    * Reject leading // in property graph source and deployment target URIs
    
    * Accept datasets as an alias for entities in property graph config
    
    * Validate that property graph field and metric expressions are strings in V1
    
    * Align V1 relationship rules and description folding with Semantic Ontology spec
    
    * Address review comments on property graph V1 validation and docs
    
    * Expose pages in Book fields for total_pages measure in integration test
    rafal-hawrylak authored Nov 6, 2026
    Configuration menu
    Copy the full SHA
    f1a6ef0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c846912 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2d6675b View commit details
    Browse the repository at this point in the history
Loading
⌂ 换网址