Skip to content

[custom_ti] Add support for basic authentication - #11202

Merged
chemamartinez merged 6 commits into
elastic:mainfrom
chemamartinez:11192-custom_ti-basic-auth
Oct 3, 2024
Merged

chemamartinez merged 6 commits into
elastic:mainfrom
chemamartinez:11192-custom_ti-basic-auth

Conversation

@chemamartinez

@chemamartinez chemamartinez commented Sep 20, 2024 •

Copy link
Copy Markdown
Contributor

Proposed commit message

This pull request adds:

  • Support for basic authentication to the Custom Threat Intelligence integration.
  • Fingerprint to avoid duplicated indicators to be ingested.

I recommend reviewing the PR commit by commit, and notice that ecfafdf only adds formatting changes (replacing tabs by spaces).

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

Screenshots

image

@chemamartinez chemamartinez added enhancement New feature or request Integration:ti_custom Custom Threat Intelligence labels Sep 20, 2024
@chemamartinez chemamartinez self-assigned this Sep 20, 2024
@nicpenning

Copy link
Copy Markdown
Contributor

I went down the road of trying to add this and test but got this error:

elastic-package test
2024/09/20 13:24:31  INFO New version is available - v0.104.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.104.0
Run test suite for the package
Run asset tests for the package
2024/09/20 13:24:31  INFO License text found in "/home/napsta/integrations/LICENSE.txt" will be included in package
Error: error running package asset tests: could not complete test run: can't install the package: can't install the package: could not zip-install package; API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"Encountered 1 errors creating saved objects: [{\"type\":\"dashboard\",\"id\":\"ti_custom-e336dd7a-d5cb-4b7f-a6cd-85c45d0bd1ac\",\"error\":{\"isBoom\":true,\"isServer\":false,\"data\":null,\"output\":{\"statusCode\":400,\"payload\":{\"message\":\"[attributes.controlGroupInput.panelsJSON]: expected value of type [string] but got [Object]: Bad Request\",\"statusCode\":400,\"error\":\"Bad Request\"},\"headers\":{}},\"type\":\"unknown\"}}]"}

Did you see this in your testing?

@nicpenning

Copy link
Copy Markdown
Contributor

👀

@chemamartinez
chemamartinez force-pushed the 11192-custom_ti-basic-auth branch from 60b13d1 to ecfafdf Compare October 2, 2024 11:01
@chemamartinez

Copy link
Copy Markdown
Contributor Author

Hi @nicpenning,

I have just run the same command successfully:

% elastic-package test
2024/10/02 13:14:17  INFO New version is available - v0.104.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.104.0
Run asset tests for the package
2024/10/02 13:14:17  INFO License text found in "/Users/chemamartinez/devel/integrations/LICENSE.txt" will be included in package
--- Test results for package: ti_custom - START ---
╭───────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                                                          │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ ti_custom │             │ asset     │ dashboard ti_custom-e336dd7a-d5cb-4b7f-a6cd-85c45d0bd1ac is loaded │ PASS   │       1.25µs │
│ ti_custom │             │ asset     │ search ti_custom-a06e63dc-01d3-4005-b24f-2fd46c3962b1 is loaded    │ PASS   │        417ns │
│ ti_custom │ indicator   │ asset     │ index_template logs-ti_custom.indicator is loaded                  │ PASS   │        334ns │
│ ti_custom │ indicator   │ asset     │ ingest_pipeline logs-ti_custom.indicator-0.2.0 is loaded           │ PASS   │        250ns │
╰───────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_custom - END   ---
Done
Run pipeline tests for the package
--- Test results for package: ti_custom - START ---
╭───────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                                                             │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-asn-ndjson.log)              │ PASS   │  214.81975ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-domain-name-ndjson.log)      │ PASS   │ 201.276292ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-email-ndjson.log)            │ PASS   │  209.85875ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-file-ndjson.log)             │ PASS   │ 203.720208ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-ip-ndjson.log)               │ PASS   │ 216.358709ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-unknown-ndjson.log)          │ PASS   │ 212.144917ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-url-ndjson.log)              │ PASS   │ 216.560917ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-windows-registry-ndjson.log) │ PASS   │ 214.379583ms │
│ ti_custom │ indicator   │ pipeline  │ (ingest pipeline warnings test-indicator-x509-ndjson.log)             │ PASS   │   200.5165ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-asn-ndjson.log                                         │ PASS   │    85.3065ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-domain-name-ndjson.log                                 │ PASS   │  85.485333ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-email-ndjson.log                                       │ PASS   │  78.396541ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-file-ndjson.log                                        │ PASS   │  91.986958ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-ip-ndjson.log                                          │ PASS   │   77.96225ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-unknown-ndjson.log                                     │ PASS   │  65.644709ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-url-ndjson.log                                         │ PASS   │  64.541625ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-windows-registry-ndjson.log                            │ PASS   │  69.884042ms │
│ ti_custom │ indicator   │ pipeline  │ test-indicator-x509-ndjson.log                                        │ PASS   │ 111.277333ms │
╰───────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_custom - END   ---
Done
Run policy tests for the package
--- Test results for package: ti_custom - START ---
No test results
--- Test results for package: ti_custom - END   ---
Done
Run static tests for the package
--- Test results for package: ti_custom - START ---
╭───────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ ti_custom │ indicator   │ static    │ Verify sample_event.json │ PASS   │  76.768125ms │
╰───────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_custom - END   ---
Done
Run system tests for the package
2024/10/02 13:08:17  INFO License text found in "/Users/chemamartinez/devel/integrations/LICENSE.txt" will be included in package
2024/10/02 13:09:06  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/stix-taxii-1727867346767679000.log
2024/10/02 13:09:14  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/elastic-agent-1727867354872666000.log
2024/10/02 13:10:01  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/stix-taxii-1727867401965130000.log
2024/10/02 13:10:10  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/elastic-agent-1727867410076043000.log
2024/10/02 13:11:04  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/stix-taxii-1727867464120480000.log
2024/10/02 13:11:12  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/elastic-agent-1727867472202292000.log
2024/10/02 13:11:56  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/stix-filestream-1727867516252147000.log
2024/10/02 13:12:04  INFO Write container logs to file: /Users/chemamartinez/devel/integrations/build/container-logs/elastic-agent-1727867524453067000.log
--- Test results for package: ti_custom - START ---
╭───────────┬─────────────┬───────────┬──────────────────┬────────┬───────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME        │ RESULT │  TIME ELAPSED │
├───────────┼─────────────┼───────────┼──────────────────┼────────┼───────────────┤
│ ti_custom │ indicator   │ system    │ filestream       │ PASS   │ 34.117025291s │
│ ti_custom │ indicator   │ system    │ stix             │ PASS   │ 41.620459791s │
│ ti_custom │ indicator   │ system    │ taxii            │ PASS   │    35.843603s │
│ ti_custom │ indicator   │ system    │ taxii-basic-auth │ PASS   │ 37.779323916s │
╰───────────┴─────────────┴───────────┴──────────────────┴────────┴───────────────╯
--- Test results for package: ti_custom - END   ---
Done

Maybe there is something broken in your elastic-package installation? you can try upgrading it to the latest version.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@chemamartinez
chemamartinez marked this pull request as ready for review October 2, 2024 11:22
@chemamartinez
chemamartinez requested a review from a team as a code owner October 2, 2024 11:22
@chemamartinez chemamartinez added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Oct 2, 2024
@elasticmachine

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@nicpenning

Copy link
Copy Markdown
Contributor

Could have been, looks like yours is coming along nicely, thanks for adding this!

@chemamartinez
chemamartinez requested a review from efd6 October 3, 2024 07:58
@elasticmachine

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @chemamartinez

@elastic-sonarqube

Copy link
Copy Markdown

@chemamartinez
chemamartinez merged commit 07a9442 into elastic:main Oct 3, 2024
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package ti_custom - 0.2.0 containing this change is available at https://epr.elastic.co/search?package=ti_custom

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
This pull request adds:
- Support for basic authentication to the Custom Threat Intelligence integration.
- Fingerprint to avoid duplicated indicators to be ingested.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
This pull request adds:
- Support for basic authentication to the Custom Threat Intelligence integration.
- Fingerprint to avoid duplicated indicators to be ingested.
@chemamartinez
chemamartinez deleted the 11192-custom_ti-basic-auth branch February 6, 2025 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:ti_custom Custom Threat Intelligence Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[custom_ti] Add Basic auth to integration

4 participants

⌂ 换网址