simple_neural_network.py: Fewer forward propogations to speed tests #11013

cclauss · 2023-10-26T22:06:55Z

Describe your change:

Add an algorithm?
Fix a bug or typo in an existing algorithm?
Add or change doctests? -- Note: Please avoid changing both code and tests in a single pull request.
Documentation change?

Checklist:

tianyizheng02 · 2023-10-26T22:12:56Z

At one point I tried to decrease the number of iterations like you did, but it didn't work. 100k iterations isn't even enough for the model to converge within 1 of the true value:

=================================== FAILURES ===================================
______ [doctest] neural_network.simple_neural_network.forward_propagation ______
029 Return the value found after the forward propagation training.
030 
031     >>> res = forward_propagation(32, 100_000)  # Was 10_000_000
032     >>> res > 31 and res < 33
Expected:
    True
Got:
    False

Like I've been saying, this implementation is really suboptimal and should be rewritten.

quant12345 · 2023-10-27T11:08:04Z

@cclauss make 1 million and the tests will pass. I tried to measure the time (not in doctest) and it was 9 - 10 times faster.
res = forward_propagation(32, 1_000_000)

performance code:

"""
Forward propagation explanation:
https://towardsdatascience.com/forward-propagation-in-neural-networks-simplified-math-and-code-version-bbcfef6f9250
"""

import math
import random


# Sigmoid
def sigmoid_function(value: float, deriv: bool = False) -> float:
    """Return the sigmoid function of a float.

    >>> sigmoid_function(3.5)
    0.9706877692486436
    >>> sigmoid_function(3.5, True)
    -8.75
    """
    if deriv:
        return value * (1 - value)
    return 1 / (1 + math.exp(-value))


# Initial Value
INITIAL_VALUE = 0.02


def forward_propagation(expected: int, number_propagations: int) -> float:
    """Return the value found after the forward propagation training.

    >>> res = forward_propagation(32, 1_000_000)
    >>> res > 31 and res < 33
    True

    >>> res = forward_propagation(32, 1000)
    >>> res > 31 and res < 33
    False
    """

    # Random weight
    weight = float(2 * (random.randint(1, 100)) - 1)

    for _ in range(number_propagations):
        # Forward propagation
        layer_1 = sigmoid_function(INITIAL_VALUE * weight)
        # How much did we miss?
        layer_1_error = (expected / 100) - layer_1
        # Error delta
        layer_1_delta = layer_1_error * sigmoid_function(layer_1, True)
        # Update weight
        weight += INITIAL_VALUE * layer_1_delta

    return layer_1 * 100


if __name__ == "__main__":
    import doctest

    doctest.testmod()

    import datetime

    now = datetime.datetime.now()
    forward_propagation(32, 10000000)
    old = datetime.datetime.now() - now

    print('old_time', old)

    now = datetime.datetime.now()
    forward_propagation(32, 1000000)
    new = datetime.datetime.now() - now

    print('new_time', new)

    print('old/new', old/new)

Output:

old_time 0:00:07.447717
new_time 0:00:00.735285
old/new 10.129020719856927

quant12345 · 2023-10-27T11:26:54Z

@cclauss I just checked 200_000, it also worked. I apologize for not checking it right away.
But, if you try several times, it happens that the tests fail. I'll try to find the optimal number.

cclauss · 2023-10-27T12:30:27Z

Thanks, @quant12345 please keep trying to make it faster without becoming a flaky test.

forward_propagation(32, 500_000) # Was 10_000_000 # puts the tests are < 1sec in 4th slowest.

============================= slowest 10 durations =============================
1.32s call     web_programming/get_imdbtop.py::web_programming.get_imdbtop.get_imdb_top_movies
1.21s call     web_programming/fetch_anime_and_play.py::web_programming.fetch_anime_and_play.search_anime_episode_list
1.00s call     web_programming/fetch_anime_and_play.py::web_programming.fetch_anime_and_play.search_scraper
0.90s call     neural_network/simple_neural_network.py::neural_network.simple_neural_network.forward_propagation
0.85s call     web_programming/fetch_anime_and_play.py::web_programming.fetch_anime_and_play.get_anime_episode
0.83s call     dynamic_programming/integer_partition.py::dynamic_programming.integer_partition.partition
0.72s call     graphs/bidirectional_a_star.py::graphs.bidirectional_a_star.AStar
0.50s call     matrix/count_negative_numbers_in_sorted_matrix.py::matrix.count_negative_numbers_in_sorted_matrix.generate_large_matrix
0.37s call     matrix/count_negative_numbers_in_sorted_matrix.py::matrix.count_negative_numbers_in_sorted_matrix.count_negatives_brute_force
0.37s call     dynamic_programming/matrix_chain_multiplication.py::dynamic_programming.matrix_chain_multiplication.matrix_chain_order
================== 1749 passed, 1 warning in 72.96s (0:01:12) ==================

cclauss · 2023-10-27T12:42:17Z

@tianyizheng02 For the web_programming jobs, should we consider replacing requests with the plug-compatible httpx and encouraging contributors to use asyncio for web interactions? I doubt it would speed up the tests but it would continue to push contributors to learn new skills. We currently have 35 files that import requests.

quant12345 · 2023-10-27T12:43:48Z

Thanks, @quant12345 please keep trying to make it faster without becoming a flaky test.

forward_propagation(32, 500_000) # Was 10_000_000 # puts the tests are < 1sec in 4th slowest.

Now I tried 400_000 by calling the function 100 times, sometimes the tests fail. At 500_000 I tried 1000, all tests passed(that is, I also think 500_000 is close to the optimal number).

if __name__ == "__main__":
    import doctest

    doctest.testmod()

    for i in range(100):
        doctest.run_docstring_examples(forward_propagation, globals())

cclauss · 2023-10-27T12:45:43Z

Nice! Could you please try 450_000 before I merge this?

quant12345 · 2023-10-27T12:47:56Z

Nice! Could you please try 450_000 before I merge this?

I run tests 1000 times. It takes some time.

@cclauss

Update: ran it 1000 times with the number 450_000 tests were successful.

cclauss · 2023-10-27T21:32:38Z

LGTM. @tianyizheng02

cclauss and others added 2 commits October 27, 2023 00:02

simple_neural_network.py: Fewer forward propogations to speed tests

1b31c25

Closes #9718

updating DIRECTORY.md

6837542

algorithms-keeper bot added enhancement This PR modified some existing files awaiting reviews This PR is ready to be reviewed tests are failing Do not merge until tests pass labels Oct 26, 2023

cclauss mentioned this pull request Oct 27, 2023

Speedup our eight slowest pytests (one at a time please) #9718

Closed

8 tasks

forward_propagation(32, 1_000_000) # Was 10_000_000

ed4ec69

algorithms-keeper bot removed the tests are failing Do not merge until tests pass label Oct 27, 2023

forward_propagation(32, 500_000) # Was 10_000_000

dc7448d

Update simple_neural_network.py

8668f57

cclauss enabled auto-merge (squash) October 27, 2023 21:19

cclauss requested a review from tianyizheng02 October 27, 2023 21:19

cclauss requested a review from ChrisO345 October 28, 2023 21:28

cclauss mentioned this pull request Oct 28, 2023

Create ipv4_conversion.py #11008

Merged

15 tasks

cclauss closed this in #11008 Oct 28, 2023

auto-merge was automatically disabled October 28, 2023 22:47
Pull request was closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

simple_neural_network.py: Fewer forward propogations to speed tests #11013

simple_neural_network.py: Fewer forward propogations to speed tests #11013

cclauss commented Oct 26, 2023

tianyizheng02 commented Oct 26, 2023

quant12345 commented Oct 27, 2023 •

edited

Loading

quant12345 commented Oct 27, 2023 •

edited

Loading

cclauss commented Oct 27, 2023 •

edited

Loading

cclauss commented Oct 27, 2023 •

edited

Loading

quant12345 commented Oct 27, 2023

cclauss commented Oct 27, 2023

quant12345 commented Oct 27, 2023 •

edited

Loading

cclauss commented Oct 27, 2023

simple_neural_network.py: Fewer forward propogations to speed tests #11013

simple_neural_network.py: Fewer forward propogations to speed tests #11013

Conversation

cclauss commented Oct 26, 2023

Describe your change:

Checklist:

tianyizheng02 commented Oct 26, 2023

quant12345 commented Oct 27, 2023 • edited Loading

quant12345 commented Oct 27, 2023 • edited Loading

cclauss commented Oct 27, 2023 • edited Loading

cclauss commented Oct 27, 2023 • edited Loading

quant12345 commented Oct 27, 2023

cclauss commented Oct 27, 2023

quant12345 commented Oct 27, 2023 • edited Loading

cclauss commented Oct 27, 2023

quant12345 commented Oct 27, 2023 •

edited

Loading

quant12345 commented Oct 27, 2023 •

edited

Loading

cclauss commented Oct 27, 2023 •

edited

Loading

cclauss commented Oct 27, 2023 •

edited

Loading

quant12345 commented Oct 27, 2023 •

edited

Loading