Thank you for your question and pics.  My first thought is that the villians are aphids.  The article I found from Clemson.edu contains other info as well, but their description of aphids seem to match your pics.  The article does contain information about treatment, but I did not include it because you indicated you do not use chemicals.  I hope this info helps.  I would be looking for further evidence on the leaves as well.  The second article is from the American Daylily Society and may be of help as well.



https://hgic.clemson.edu/factsheet/daylily-diseases-insect-pests/

Insects & Other Pests

Flower Thrips (Frankliniella tritici): Flower thrips and various other thrips species are serious pests of daylilies. Thrips are slender, dark-colored insects with fringed wings. Adults are less than ¹/₁₆-inch in length. To see these small, fast-moving pests, use a 10x magnifying lens. Thrips are typically found on leaves and between flower petals. Both adults and nymphs (immature insect stage resembling the adult, but smaller) feed by scraping surface cells to suck plant sap. When they feed on flower buds, the flower may die without opening. With a light infestation, their feeding causes leaves to have silvery speckles or streaks. With severe infestations, leaves and flowers are stunted and distorted and may turn brown and die.

Sampling: As a result of their small size, thrips are difficult to detect before damage is obvious. To sample for thrips on daylilies, hold a sheet of stiff white paper under some leaves and flowers, and then strike the paper with these plant parts. Gently tip the paper to remove any bits of trash and then examine the paper in bright sunlight. Any thrips present will move around on the paper.

Control: Several naturally occurring enemies feed on thrips. To avoid killing these beneficial insects, which naturally reduce thrips populations, insecticides should be avoided as much as possible. Blue sticky traps will help protect daylilies from thrips. Paint cardboard or wooden boards blue and then coat them with petroleum jelly. Attach them to stakes and place them near the daylilies.

If serious damage is occurring, insecticidal soap or spinosad sprays are recommended. Spinosad is a foliar systemic insecticide that can penetrate the unopened flowers to give good thrips control. Always spray in the evening to lessen the spray impact on pollinating insects. As with any pesticide, read and follow all label directions and precautions.

Two-spotted Spider Mites (Tetranychus urticae): The two-spotted spider mite and other mite species can be a problem on daylilies. Mites are not insects but are more closely related to spiders. They are very small, less than ¹/₅₀-inch long. They have piercing mouthparts that allow them to puncture plant tissue and suck plant sap. Mites tend to be more of a problem during hot, dry periods. Over time, some spider mites produce a fine web on leaves, which protects their eggs and young. With a light infestation, daylily leaves and flowers develop yellowish speckles. Partially because of the mite’s tiny size, this damage often goes unnoticed until the damage is more severe. With a heavy infestation, the speckles will run together, and entire leaves can become bleached and die. Along with leaf decline, growth is stunted.

Sampling: Like thrips, spider mites are very small and are difficult to detect before damage is obvious. To sample for mites on daylilies, follow the same procedure discussed in the thrips section above.

Control: Spider mites overwinter (survive the winter) on weeds, such as chickweed. Removing nearby weeds before spring growth is an important step in the control of spider mites. Insecticidal soap spray, if started early in the infestation, is effective at controlling spider mites. For more severe infestations, tau-fluvalinate or bifenthrin sprays will control spider mites. See Table 1 for examples of brands and products. As with all pesticides, read and follow all label directions and precautions.

Aphids: Various aphids can be pests on daylilies. Aphids are soft-bodied insects that vary in color from yellow-green to almost black. They are typically more of a problem during cool weather in the spring. They feed on leaves and flower buds by inserting their mouthparts and sucking plant sap. Their feeding can result in deformed leaves and small warty growths on flower buds. Most aphids excrete honeydew (a sugary liquid waste) after feeding on plant sap. A fungus called sooty mold will grow on the honeydew and result in dark fungal growth on the foliage.

Control: Several naturally occurring enemies feed on aphids, including green lacewings and ladybird beetles (ladybugs). As much as possible, these predators should be allowed to reduce aphid populations. Because of their phenomenal reproductive rate, aphids are very difficult to control with insecticides. If a single aphid survives, a new colony can be produced within a short time. In addition, the use of contact insecticides may also kill the beneficial predators of aphids. Insecticidal soap sprays are a less toxic alternative that can help reduce aphid populations.

However, for more severe infestations, bifenthrin, cyfluthrin, lambda cyhalothrin, or permethrin sprays will control aphids. See Table 1 for examples of brands and products.

Slugs & Snails: These can be a problem on daylilies, especially in the early spring when they feed on tender young growth. Their feeding results in ragged notches along leaf edges and sometimes holes in the middle of leaves. The appearance of shiny, slimy trails is a typical sign of their presence. Slugs and snails feed at night and hide during the day in moist areas.

Control: Remove their daytime hiding places by removing mulch and leaf litter near the base of the plant. Apply environmentally safe snail and slug baits (containing iron phosphate) near the beds. More information on control of slugs and snails and the safest baits is available in HGIC 2357, Snails & Slugs in the Home Garden.

Daylily leafminer (Ophiomyia kwansonis) damage is more aesthetic but not extremely significant to plant health.
Joey Williamson, 2015 HGIC, Clemson Extension

Daylily Leafminer: The daylily leafminer (Ophiomyia kwansonis) was introduced into the US around 2006 from Japan and Taiwan and has quickly spread over the Eastern US. The leafminer adult is a small black fly, which oviposits (lays) its eggs on daylily foliage in the spring. The small pale-yellow larvae then feed within the leaves in meandering mines that become wider as the larvae grow. The whitish mines remain visible throughout the life of the leaf, which makes infested daylilies unmarketable. This pest over-winters as small tan pupae, approximately the size of a grain of rice. Over-wintering occurs either near the base of the leaves or at the base of the plants.

Control: Prompt removal and disposal of infested foliage will reduce the spread of this pest. Clean up and dispose of all dead foliage at the end of the season, as this will help reduce over-wintering of the pupae, as well as reduce the spread of foliar diseases, such as daylily leaf streak and daylily rust. Sprays to control the leafminer may affect natural predators and parasites that can reduce the number of leafminers.

Insecticides that control this leafminer are imidacloprid sprays or plant drenches, acephate sprays, or spinosad sprays. Spinosad is a natural product that is the safest to apply, and like acephate, is a foliar systemic insecticide. This means it will penetrate the foliage to kill the larvae within the leaves. Spinosad is used to control insect pests, including thrips, caterpillars, and flies, so it is less harmful to many beneficials. However, it is always best to spray late in the evening to reduce the impact on pollinating insects. Spray with spinosad as soon as symptoms of mining are noticed.

https://daylilies.org/daylily-dictionary/aphids/

Aphids are small, mostly less then one tenth of an inch, soft-bodied insects of the family Aphididae. The daylily aphid, Myzus hemerocallis, is, not surprisingly, often found on daylilies but there are many other kinds of aphids; some of which may also affect daylilies from time to time. The daylily aphid is light green in color and feeds in colonies between the leaves of the plant, inserting the mouth parts into plant tissue and sucking up the juices. Whitish specks are often noticed on the plants when aphids are present. These are the cast skins left behind after molting. (See lower image below.) They also excrete a sticky substance called “honeydew”. The image accompanying this text is of the related green peach aphid (Myzus persicae). The life cycle of aphids can be rather complex and can vary somewhat according to species and geographical region. In mild weather aphids can produce continuous generations rapidly by live birth without mating, most of the population being female. With the imminent onset of cold weather in Fall, males may be produced, mating occurs and the females lay overwintering eggs. In warm climates and in greenhouses, some species may omit the egg-laying stage and continue to reproduce by live birth through the winter, although where it is slightly colder they may hibernate as adults. Most aphids are wingless, but sometimes you will see winged forms. Natural enemies may keep populations down to levels where no treatment is necessary. Aphids are especially partial to plants fertilized with large amounts of nitrogen. If control is desired, insecticidal soap can be effective if the aphids are located where they can be directly hit by the spray. Numerous other low or non-toxic remedies are discussed in various references including repellent companion plants such as the allium (onion) family, coriander and anise.

Photograph: Used with permission from University of California Statewide IPM Project, Jack Kelly Clark photographer. Any additional uses of this image must be received from the copyright holder directly – AHS does not have permission to grant additional usage of this image for any purposes.

Aphid’s cast skins, left after molting, are shown in the closeup.- Photo courtesy of Sue Bergeron, used with permission.

I am not familiar with neem oil.  I found a comprehensive fact sheet from Oregon State about it.  I hope this helps.
http://npic.orst.edu/factsheets/neemgen.html

Neem Oil

General Fact Sheet

• What is neem oil?
• What are some products that contain neem oil?
• How does neem oil work?
• How might I be exposed to neem oil?
• What are some signs and symptoms from a brief exposure to neem oil?
• What happens to neem oil when it enters the body?
• Is neem oil likely to contribute to the development of cancer?
• Has anyone studied non-cancer effects from long-term exposure to neem oil?
• Are children more sensitive to neem oil than adults?
• What happens to neem oil in the environment?
• Can neem oil affect birds, fish, and other wildlife?

What is neem oil?

Neem oil is a naturally occurring pesticide found in seeds from the neem tree. It is yellow to brown, has a bitter taste, and a garlic/sulfur smell. It has been used for hundreds of years to control pests and diseases. Components of neem oil can be found in many products today. These include toothpaste, cosmetics, soaps, and pet shampoos. Neem oil is a mixture of components. Azadirachtin is the most active component for repelling and killing pests and can be extracted from neem oil. The portion left over is called clarified hydrophobic neem oil.

What are some products that contain neem oil?

Neem oil and some of its purified components are used in over 100 pesticide products. They are applied to a wide variety of crops and ornamental plants for insect control. Neem oil can be formulated into granules, dust, wettable powders or emulsifiable concentrates.

Always follow label instructions and take steps to avoid exposure. If any exposures occur, be sure to follow the First Aid instructions on the product label carefully. For additional treatment advice, contact the Poison Control Center at 1-800-222-1222. If you wish to discuss a pesticide problem, please call 1-800-858-7378.

How does neem oil work?

Neem oil is made of many components. Azadirachtin is the most active. It reduces insect feeding and acts as a repellent. It also interferes with insect hormone systems, making it harder for insects to grow and lay eggs. Azadirachtin can also repel and reduce the feeding of nematodes. Other components of neem oil kill insects by hindering their ability to feed. However, the exact role of every component is not known.

How might I be exposed to neem oil?

People can be exposed to chemicals by eating them, breathing them in, through skin contact and eye contact. Since neem oil is used on a variety of crops, people are mainly exposed to neem oil in their diet. People who apply neem oil may also be exposed if they inhale the mist or dust, let the product touch their skin, or fail to wash their hands before eating or smoking. However, the label includes directions for keeping exposure low. For example the label might require applicators to wear protective clothing.

What are some signs and symptoms from a brief exposure to neem oil?

Neem oil can be slightly irritating to the eyes and skin. Azadirachtin, a component of neem oil, can be very irritating to the skin and stomach. The remaining portion of neem oil is made of fatty acids, essential oils and other substances that are commonly eaten in a normal diet. These substances are generally recognized as safe (GRAS) by the United States Food and Drug Administration.

In other countries, neem oil has been used on cats for flea control. Some adverse reactions have been reported. Symptoms include feeling sluggish, excessive salivation, impaired movement, trembling, twitching, and convulsions. Some of the cats died. However, most of them recovered within 1 to 5 days.

What happens to neem oil when it enters the body?

Clarified hydrophobic neem oil (without azadirachtin) is made of fatty acids and glycerides. These substances are commonly found in food. When they enter the body, they are broken down, used for energy, and incorporated into cells.

In one study, scientists injected insects with azadirachtin. They found 90% of the dose in the insects' feces within 7 hours. The remaining portion lingered in the insects' bodies for 24 days after the injection.

Is neem oil likely to contribute to the development of cancer?

No. People have been exposed to neem oil in many ways for hundreds of years. During this time no association with increased cancer risk has been found. Studies showed that neem oil did not alter or damage genes. In laboratory tests, animals were fed neem oil for 90 days. They did not have increased cancer rates.

Further, one study found that certain components of neem oil caused cancer cells in hamsters to stop growing or die. Another study looked at prostate cancer cells from humans. Researchers found that neem leaf extract was able to slow their growth.

Has anyone studied non-cancer effects from long-term exposure to neem oil?

In rat studies, no effects were reported when the rats were fed either azadirachtin or clarified hydrophobic neem oil throughout their lives.

Are children more sensitive to neem oil than adults?

In general, children may be especially sensitive to pesticides compared to adults. When rats were fed neem oil in one study, their pregnancies ended. In another study, rats were fed azadirachtin in their diet throughout their lives. No effects to their offspring were found. Additionally, neem oil is used in toothpaste, cosmetics, soaps and traditional medicines around the world. Therefore, people of all ages are commonly exposed to neem oil. No data were found to show that children are more sensitive than adults to neem oil.

What happens to neem oil in the environment?

Azadirachtin, a major component of neem oil, is rapidly broken down. Microbes and light break down the pesticide in soil, water and on plants. The half-life of azadirachtin in soil ranges from 3 - 44 days. In water, the half-life ranges from 48 minutes to 4 days. It also rapidly breaks down on plant leaves; the half-life is 1 - 2.5 days. The remaining components of neem oil are broken down by microbes in most soil and water environments.

Can neem oil affect birds, fish, or other wildlife?

Neem oil is practically non-toxic to birds, mammals, bees and plants. Neem oil is slightly toxic to fish and other aquatic organisms. Azadirachtin, a component of neem oil, is moderately toxic to fish and other aquatic animals. It is important to remember that insects must eat the treated plant to be killed. Therefore, bees and other pollinators are not likely to be harmed.

Where can I get more information?

For more detailed information about neem oil please visit the list of referenced resources or call the National Pesticide, Monday - Friday, between 8:00am - 12:00pm Pacific Time (11:00am - 3:00pm Eastern Time) at 1-800-858-7378 or visit us on the web at http://npic.orst.edu. NPIC provides objective, science-based answers to questions about pesticides.

Date Reviewed: March 2012

Please cite as: Bond, C.; Buhl, K.; Stone, D. 2012. Neem Oil General Fact Sheet; National Pesticide Information Center, Oregon State University Extension Services. http://npic.orst.edu/factsheets/neemgen.html.

NPIC fact sheets are designed to answer questions that are commonly asked by the general public about pesticides that are regulated by the U.S. Environmental Protection Agency (U.S. EPA). This document is intended to be educational in nature and helpful to consumers for making decisions about pesticide use.

Please read our disclaimer | Contact us | About NPIC | En español

NPIC provides objective, science-based information about pesticides and pesticide-related topics to enable people to make informed decisions. NPIC is a cooperative agreement between Oregon State University and the U.S. Environmental Protection Agency (cooperative agreement #X8-83947901). The information in this publication does not in any way replace or supersede the restrictions, precautions, directions, or other information on the pesticide label or any other regulatory requirements, nor does it necessarily reflect the position of the U.S

I think this picture matches your pic.  This is a responce written  by another master gardener in 219.  I have included the source.
https://ask2.extension.org/kb/faq.php?id=594090

Dark Green Slimy Fungus? #594090

Asked September 03, 2019, 12:10 PM EDT

We have a gravel driveway to our barn. There is some kind of fungus, I believe, that is growing in the gravel. When the weather is dry the fungus is black and crusty, but when it rains, it turns into this dark green slimy, slippery stuff. Do you know what this is and how to get rid of the stuff?

County Michigan

Expert Response

Hello and thank you for using eXtension for your " green slim" problem. What I believe you have it Nostoc Algae. It is a bit difficult to control but there is some that say baking soda works. Adequate drainage is the most important step to take in controlling it. Here is a link to an article for more information about it. https://www.canr.msu.edu/news/nostoc_a_green_jelly_like_substance_growing_in_lawns
Warm regards,
Beth Clawson Replied September 04, 2019, 11:27 AM EDT

I am glad I could help.