You can import the logging module into your code and write to logs that way

    import logging

    logging.info(&#39;Hello&#39;)

Here are some more options

    import logging    

    logging.debug(&#39;This is a debug message&#39;)
    logging.info(&#39;This is an info message&#39;)
    logging.warning(&#39;This is a warning message&#39;)
    logging.error(&#39;This is an error message&#39;)
    logging.critical(&#39;This is a critical message&#39;)

You might use airflow logger 


```
import logging


logger = logging.getLogger(&quot;airflow.task&quot;)
logger.error(&quot;Your custom error&quot;)
```


There&#39;s a logger mixin class you can use:

https://github.com/apache/airflow/blob/main/airflow/utils/log/logging_mixin.py

    from airflow.utils.log.logging_mixin import LoggingMixin
    
    LoggingMixin().log.info(&quot;Hello&quot;)

Then in airflow logs you&#39;ll see:

    [2021-07-07 15:55:42,370] {{logging_mixin.py:112}} INFO - Hello

I try to figure out where to best store application production secrets for an ASP.NET Core app. There are two similar questions
https://stackoverflow.com/questions/31426358/where-should-i-store-the-connection-string-for-the-production-environment-of-my
and
https://stackoverflow.com/questions/39668456/how-to-deploy-asp-net-core-usersecrets-to-production
which both recommend using environment variables. 

My problem is that I want to run several instances of my web app with different databases and different database credentials. So there should be some per-instance configuration including secrets.

How could this be achieved in a safe way?

Note that the application should be able to self-host and be hostable under IIS! (Later we also plan to run it on Linux if that is of any importance for the question)

**Update**

This question is not about trying to use ASP.NET user secrets in production! UserSecrets are ruled out for production.

Storing production secrets in ASP.NET Core

I want to use some useful functions as commands. For that I am testing the `click` library. I defined my three original functions then decorated as `click.command`:

    import click
    import os, sys
    
    @click.command()
    @click.argument(&#39;content&#39;, required=False)
    @click.option(&#39;--to_stdout&#39;, default=True)
    def add_name(content, to_stdout=False):
        if not content:
            content = &#39;&#39;.join(sys.stdin.readlines())
        result = content + &quot;\n\tadded name&quot;
        if to_stdout is True:
            sys.stdout.writelines(result)
        return result
    
    
    @click.command()
    @click.argument(&#39;content&#39;, required=False)
    @click.option(&#39;--to_stdout&#39;, default=True)
    def add_surname(content, to_stdout=False):
        if not content:
            content = &#39;&#39;.join(sys.stdin.readlines())
        result = content + &quot;\n\tadded surname&quot;
        if to_stdout is True:
            sys.stdout.writelines(result)
        return result
    
    @click.command()
    @click.argument(&#39;content&#39;, required=False)
    @click.option(&#39;--to_stdout&#39;, default=False)
    def add_name_and_surname(content, to_stdout=False):
        result = add_surname(add_name(content))
        if to_stdout is True:
            sys.stdout.writelines(result)
        return result

This way I am able to generate the three commands `add_name`, `add_surname` and `add_name_and_surname` using a `setup.py` file and `pip install --editable .` Then I am able to pipe:

    $ echo &quot;original content&quot; | add_name | add_surname 
    original content
    
        added name
        added surname

However there is one slight problem I need to solve, when composing with different click commands as functions:

    $echo &quot;original content&quot; | add_name_and_surname 
    Usage: add_name_and_surname [OPTIONS] [CONTENT]
    
    Error: Got unexpected extra arguments (r i g i n a l   c o n t e n t 
    )

I have no clue why it does not work, I need this `add_name_and_surname` command to call `add_name` and `add_surname` not as command but as functions, else it defeats my original purpose of using functions as both  library functions and commands. 


Call another click command from a click command

One way to write to the logs in Airflow is to return a string from a PythonOperator like on line 44 [here][1].

Are there other ways that allow me to write to the airflow log files? I&#39;ve found that print statements are not saved to the logs.

  [1]: https://github.com/apache/incubator-airflow/blob/ff45d8f2218a8da9328161aa66d004c3db3b367e/airflow/example_dags/example_python_operator.py

Writing to Airflow Logs

<p>One way to write to the logs in Airflow is to return a string from a PythonOperator like on line 44 <a href="https://github.com/apache/incubator-airflow/blob/ff45d8f2218a8da9328161aa66d004c3db3b367e/airflow/example_dags/example_python_operator.py" target="_blank" rel="noopener noreferrer">here</a>.</p>
<p>Are there other ways that allow me to write to the airflow log files? I've found that print statements are not saved to the logs.</p>


I have started the Airflow webserver and scheduled some dags. I can see the dags on web GUI.

How can I delete a particular DAG from being run and shown in web GUI? Is there an Airflow CLI command to do that?

I looked around but could not find an answer for a simple way of deleting a DAG once it has been loaded and scheduled.

Airflow: how to delete a DAG?

## Problem

Is there any way in Airflow to create a workflow such that the number of tasks B.* is unknown until completion of Task A? I have looked at subdags but it looks like it can only work with a static set of tasks that have to be determined at Dag creation.

Would dag triggers work? And if so could you please provide an example.

I have an issue where it is impossible to know the number of task B&#39;s that will be needed to calculate Task C until Task A has been completed. Each Task B.* will take several hours to compute and cannot be combined.

                  |---&gt; Task B.1 --|
                  |---&gt; Task B.2 --|
     Task A ------|---&gt; Task B.3 --|-----&gt; Task C
                  |       ....     |
                  |---&gt; Task B.N --|
                   
## Idea #1

I don&#39;t like this solution because I have to create a blocking ExternalTaskSensor and all the Task B.* will take between 2-24 hours to complete. So I do not consider this a viable solution. Surely there is an easier way? Or was Airflow not designed for this?

    Dag 1
    Task A -&gt; TriggerDagRunOperator(Dag 2) -&gt; ExternalTaskSensor(Dag 2, Task Dummy B) -&gt; Task C

    Dag 2 (Dynamically created DAG though python_callable in TriggerDagrunOperator)
                   |-- Task B.1 --|
                   |-- Task B.2 --|
    Task Dummy A --|-- Task B.3 --|-----&gt; Task Dummy B
                   |     ....     |
                   |-- Task B.N --|

Edit 1:
-------

**As of now this question still does not have a great answer**. I have been contacted by several people looking for a solution. 

Proper way to create dynamic workflows in Airflow

when I put a new DAG python script in the dags folder, I can view a new entry of DAG in the DAG UI but it was not enabled automatically. On top of that, it seems does not loaded properly as well. I can only click on the Refresh button few times on the right side of the list and toggle the on/off button on the left side of the list to be able to schedule the DAG. These are manual process as I need to trigger something even though the DAG Script was put inside the dag folder. 

Anyone can help me on this ? Did I missed something ? Or this is a correct behavior in airflow ?

By the way, as mentioned in the post title, there is an indicator with this message &quot;This DAG isn&#39;t available in the webserver DagBag object. It shows up in this list because the scheduler marked it as active in the metdata database&quot; tagged with the DAG title before i trigger all this manual process.

Airflow &quot;This DAG isnt available in the webserver DagBag object &quot;

I am using a **LocalExecutor** and my dag has **3 tasks** where task(C) is dependant on task(A). Task(B) and task(A) can run in parallel something like below

A--&gt;C

B

**So task(A) has failed** and but **task(B) ran fine**. Task(C) is yet to run as task(A) has failed.

My question is **how do i re run Task(A) alone so Task(C) runs** once Task(A) completes and Airflow UI marks them as success.



How to restart a failed task on Airflow

I am a new user of Airbnb&#39;s open source workflow/datapipeline software [airflow]. There are dozens of default example dags after the web UI is started.  I tried many ways to remove these dags, but I&#39;ve failed to do so. 

* `load_examples = False` is set in airflow.cfg.
* Folder lib/python2.7/site-packages/airflow/example_dags is removed.

States of those example dags are changed to gray after I removed the dags folder, but the items still occupy the web UI screen. And a new dag folder is specified in airflow.cfg as `dags_folder = /mnt/dag/1`. I checked this dag folder, nothing is there. It&#39;s really weird to me why it is so difficult to remove these examples.

[airflow]: https://github.com/apache/incubator-airflow

Content Type	Original Author	Original Content on Stackoverflow
Question	thatkaiguy	View Question on Stackoverflow
Solution 1 - Airflow	J.Fratzke	View Answer on Stackoverflow
Solution 2 - Airflow	Heedge	View Answer on Stackoverflow
Solution 3 - Airflow	Alejandro Kaspar	View Answer on Stackoverflow

Writing to Airflow Logs

Airflow Problem Overview

Airflow Solutions

Solution 1 - Airflow

Solution 2 - Airflow

Solution 3 - Airflow

Call another click command from a click command

Storing production secrets in ASP.NET Core

Attributions