Machine Learning For Web: Final

Concept:

This project uses a webcam and hand gestures to control a character in a Unity WebGL environment.

Behind the Concept

Throughout my time here at ITP, I have spent a lot of time in one particular software platform: Unity.  I jokingly tell my friends that I look at the Unity skybox more often than I look at the actual sky (which unfortunately is not too far away from the truth).  When we had the presentation by the Google team about their teachable machine.  The ease of training a model appealed to me and learning of how to integrate this into Unity came up as a possible application.  I had no experience publishing a unity game to web and had only tried the teachable machine once before attempting to make this project.  I definitely ran into some hurdles along the way but I am certainly satisfied with the final result and can’t wait to apply this in other ways!

Process:

Building the Unity WebGL Game

For the game and assets I downloaded asset pack from the unity asset store (https://assetstore.unity.com/packages/templates/packs/low-poly-game-kit-110455) which provided a base for me to build off of.  This part was not too tricky; the difficulty came in trying to understand the player controller script and modifying it in order to receive custom inputs. Once that was accomplished, it was a matter of creating functions for these inputs that can be accessed by my javascript sketch and publishing it to web.

 

Training Google Image Classifier

Training the model is very easy, but making sure your data is as clean (clear background, adjust hand gestures within class to account for margin of error).  Following is an example and not the training set used in final game.

 

 

Integrating Unity and Javascript

This part was definitely the hardest part technically.  There is a method you can call from javascript on the Unity game instance called “SendMessage()” which receives the gameObject, the function you want to call, and the value to pass into that function.  However, this can only be called once the game has fully loaded.  Identifying this point was difficult so, with some advice from Yining, I added a start button that would tell the Javascript sketch to run only after that button had been pressed.

Demo: https://dylandawk.github.io/ml4w-homework/FinalTest03/

Github: https://github.com/dylandawk/ml4w-homework/tree/master/FinalTest03

 

Advertisements

Duet Machine

Problem: I love to sing, and some of my favorite songs to sing are duets or have multiple parts.  But, because I am so alone and usually sing by myself, I have no one to sing these wonderful duets with.

Solution:  Since I don’t like going out and meeting people in ‘real life’, I decided to use the principles of machine learning to train a model how to recognize when I am singing certain parts of a song.  In this example, I am using the opening lines from The Lion King hit “Can You Feel the Love Tonight”.  It is a simple example, but I hope to expand it to include the longer parts of songs.

 

 

 

Magic Windows: Final Project

Screen Shot 2018-12-20 at 2.18.33 PM

There are hundreds of parks in New York City alone and each park adds a bit of reprise from the concrete jungle and a sense of community in whichever neighborhood it is located.

For my final project I wanted to utilize ARKit to create an app that would encourage users to go outside, visit their local parks and interact with the space virtually.  The how of the app was to allow users to grow virtual gardens in the parks. in order to do this I needed to accomplish a couple of objectives.

  1. Determine user’s location and whether or not they are in a park
  2. Create AR functionality that allows users to plant crops when they are in the park

User Location

In order to find the user’s location, I used Unity’s Location Services to return the current longitude and latitude of the user’s device. Secondly, I used Google’s Places API to conduct a nearby search to determine whether or not the user is nearby a park.

Screen Shot 2018-12-20 at 2.33.37 PM.png
Here is a screenshot of the app where I manually entered the longitude and latitude of Central Park which was sent to Google Places API.

 

 

AR Functionality

In order to build the functioning app I used Unity’s ARKit Plugin which allowed me to find planes on the floor or any other flat surface.  I imported crop models from the Unity asset store and built the app so that when the API indicated that the user was nearby a park, the user could touch the screen and a plant would appear in AR space.  Additionally, I made it so that the crop changed for different park the user was next to.

 

The X-Code build and scripts I created can be found here.

ICM/PCOMP: Final Project “Deep Dive” (Formerly: “Escape the Cave”)

3D Model of Setup

Project Summary: “Deep Dive” is an interactive “escape” game that combines both physical inputs and digital visuals.  Players must walk around a 24 square-foot mat to discover the buttons hidden underneath as well as use a handheld button in order to navigate through an underwater cave and make it to the surface before the time runs out.

Project Components:

  • Software:
    • Unity Engine
    • Arduino
  • Hardware:
    • 24 sqft foam mat
    • x4 aluminum switches
    • x1 push-button attached to goggles
    • Projector
    • Wiring and perf-board
    • Arduino Uno

Development:

Initially, we developed the basic functionality of the project and prototyped it.  In this example testers would step on a yoga mat and I would manually press the left or right buttons on my computer if they stepped on the right spot:

Further Development and Objectives: ICM Presentation

By this point the primary functionality had been programmed and additional code was written to add supplementary elements such as a UI and audio.

Challenges;

  • Navigating Unity. Using Unity as a game engine is not new to me, but this was my first full-fledged game and a lot of the functionality needed to be learned outside of class. This included:
    • First-Person Character Controllers
    • Serial Communication in Unity
    • Texturing
    • IENumerators
    • External Script referencing
    • Animation
    • Audio
    • Timers

Code:

The code for the game can be found here: https://github.com/dylandawk/PCOMP-ICM-Final/tree/master/Scripts

The raw code may be a bit difficult to decipher because of the lack of commenting on my part (sorry) and because of the nature of how all the parts connected within the project is omitted.

Updated Demo:

Updated screen recording includes On-screen instructions, timer, and final 3D models (audio not available for screen recording).

 

 

 

 

PCOMP/ICM Final: Escape the Cave (Code)

Source Code
Player Controller:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.IO.Ports;

/// <summary>
/// THe script defines how the user serial input affects the game main references
/// for this code com efrom Unity Documentation and Unity Forum
/// link: https://docs.unity3d.com/ScriptReference/
/// link: https://answers.unity.com/index.html
/// </summary>

public class CameraControl : MonoBehaviour
{

    public Camera cam1;
    public Camera cam2;

    private static string port = "/dev/cu.usbmodem146401";
    private static int baudRate = 9600;
    SerialPort sp = new SerialPort(port, baudRate);
    public IsVisible lightVis;
    public IsVisible tankVis;
    public IsVisible dolphinVis;
    public GameObject dolphin;
    public bool interactPressed;
    private float t;
    private float speed = 2f;
    private bool gameStarted;
    private Rigidbody player;

    public LightControl light;
    public LightControl tank;
    public GameObject spotlight;
    public Transform spotlightPrefab;
    private Quaternion spotlightRot; 
    private Vector3 centerPos;
    private Vector3 lastPos;
    private Quaternion lastRot;
    private bool lightAcquired;
    private bool airAcquired;
    private bool dolphinInstantiated;

    private bool isRunning;


    IEnumerator FallThrough(){
        var fromPos = transform.position;
        var toPos = new Vector3(0.012f, -7f, -9f);
        for (var i = 0f; i = closeEnough)
        {
            // Confirm that it's moving
            //Debug.Log("Executing Movement");

            // Move a bit then  wait until next  frame
            transform.position = Vector3.Slerp(transform.position, targetPos, delta);
            transform.rotation = Quaternion.Lerp(transform.rotation, targetRot, delta);
            yield return wait;

            // Check if we should repeat
            distance = (transform.position - targetPos).magnitude;
        }

        // Complete the motion to prevent negligible sliding
        transform.position = targetPos;
        transform.rotation = targetRot;



        // Confirm  it's ended
        isRunning = false;
        //Debug.Log("Movement Complete");
    }




    // Use this for initialization
    void Start()
    {

        cam1.enabled = true;
        cam2.enabled = false;
        interactPressed = false;
        isRunning = false;
        gameStarted = false;
        lightAcquired = false;
        airAcquired = false;
        dolphinInstantiated = false;
        dolphin.SetActive(false);
        centerPos = new Vector3(0.09f, -7f, -9.5f);
        spotlightRot = Quaternion.Euler(.62f, 80.22f, 0);

        Physics.gravity = new Vector3(0f, -1f, 0f);
        player = GetComponent();

        sp.Open();
        sp.ReadTimeout = 1;


    }

    // Update is called once per frame
    void Update()
    {

        if (sp.IsOpen)
        {

            try
            {
                RotateView(sp.ReadByte());

            }
            catch (System.Exception)
            {

            }
        }

        KeyInput();

    }

    private void KeyInput(){

        if (Input.GetKey(KeyCode.DownArrow))
        {
            transform.position -= transform.forward * speed * Time.deltaTime;
            Debug.Log("Move Backwards");
        }
        if (Input.GetKey(KeyCode.UpArrow))
        {
            transform.position += transform.forward * speed * Time.deltaTime;
            Debug.Log("Move Forwards");

        }
        if (Input.GetKey(KeyCode.Space))
        {
            cam1.enabled = false;
            cam2.enabled = true;
            sp.Close();
        }
    }

    void RotateView(int button)
    {

        Vector3 byAngles;
        float inTime = 4f;
        if(gameStarted == false){
            player.useGravity = true;
            gameStarted = true;

        }

        if (button == 0 &amp;&amp; !interactPressed)
        {
            byAngles = Vector3.up * 5;
            var fromAngle = transform.rotation;
            var toAngle = Quaternion.Euler(transform.eulerAngles + byAngles);
         
            transform.rotation = Quaternion.Slerp(fromAngle, toAngle, Time.deltaTime * inTime);

            //Debug.Log("right came through");

        }

        if (button == 1 &amp;&amp; !interactPressed)
        {
            byAngles = Vector3.up * -5;
            var fromAngle = transform.rotation;
            var toAngle = Quaternion.Euler(transform.eulerAngles + byAngles);

            transform.rotation = Quaternion.Slerp(fromAngle, toAngle, Time.deltaTime * inTime);

            //Debug.Log("left came through");

        }

        if (button == 2 &amp;&amp; !interactPressed &amp;&amp; !isRunning) {

            lastPos = transform.position;
            lastRot = transform.rotation;
            float delta = .005f;

            if (lightVis.isVis)
            {
                Vector3 toSpotlight = new Vector3(1.63f, -7f, -10.2f);
                StartCoroutine(MoveToObject(toSpotlight, spotlightRot, delta));
            }

            if (tankVis.isVis &amp;&amp; lightAcquired)
            {
                Vector3 toTank = new Vector3(-2f, -6.9f, -9.6f);
                Quaternion tankRot = Quaternion.Euler(0f, 322.35f, 0f);
                StartCoroutine(MoveToObject(toTank, tankRot, delta));
            }

            interactPressed = true;
            Debug.Log("interact pressed = true");

        
        }

        if (button == 3 &amp;&amp; interactPressed &amp;&amp; !isRunning)
        {
            float delta = .005f;
            StartCoroutine(MoveToObject(lastPos, lastRot, delta));
            interactPressed = false;
            Debug.Log("interact pressed = false");


        }


        if (button == 4)
        {

            Debug.Log("button 5 pressed");
            Debug.Log(dolphinVis.isVis);

            if (lightVis.isVis &amp;&amp; light.inRange &amp;&amp; !lightAcquired)
            {
                GameObject newlight = Instantiate(spotlight, transform.position, transform.rotation);
                newlight.transform.parent = this.transform;
                lightAcquired = true;
            } 
            else if (tankVis.isVis &amp;&amp; tank.inRange &amp;&amp; lightAcquired &amp;&amp; !dolphinInstantiated)
            {
                Debug.Log("Oxygen Acquired");
                airAcquired = true;
                //.Instantiate(dolphin,new Vector3 (-0.25f,-6.92f,-7.22f), Quaternion.identity);
                dolphin.SetActive(true);
                dolphinInstantiated = true;
            }
            else if (dolphinVis.isVis == true)
            {
                cam1.enabled = false;
                cam2.enabled = true;
                sp.Close();
                Debug.Log("Dolphin is Vis");
            }

        }



    }
}

Underwater Fog

using UnityEngine;
using System.Collections;

public class Underwater : MonoBehaviour
{
    public float waterHeight;
    private bool isUnderwater;
    private Color normalColor;
    private Color underwaterColor;

    // Use this for initialization


    void Start()
    {
        normalColor = new Color(0.5f, 0.5f, 0.5f, 0.5f);
        underwaterColor = new Color(.165f,0.2f,0.29f, .5f);
        waterHeight = 0;

    }

    // Update is called once per frame
    void Update()
    {

        if ((transform.position.y < waterHeight) != isUnderwater)
        {
            isUnderwater = transform.position.y < waterHeight;
            if (isUnderwater) SetUnderwater();
            if (!isUnderwater) SetNormal();

        }


    }

    void SetNormal()
    {
        RenderSettings.fogColor = normalColor;
        RenderSettings.fogDensity = 0.01f;
        Physics.gravity = new Vector3(0f, -1f, 0f);

    }

    void SetUnderwater()
    {
        RenderSettings.fogColor = underwaterColor;
        RenderSettings.fogDensity = 0.7f;
        Debug.Log("true"); 
        Physics.gravity = new Vector3(0f, -.2f, 0f);

    }


}

Object Visible:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class IsVisible : MonoBehaviour
{
Renderer m_Renderer;
public bool isVis;

// Use this for initialization
void Start()
{
m_Renderer = GetComponent();
isVis = false;
}

// Update is called once per frame
void Update()
{
if (m_Renderer.isVisible)
{
//Debug.Log(“Object is visible”);
isVis = true;

}
else if(!m_Renderer.isVisible) {

//Debug.Log(“Object is no longer visible”);
isVis = false;

}
}
}

Light Control

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class LightControl : MonoBehaviour {

    public bool inRange;


	// Use this for initialization
	void Start () {

        inRange = false;
		
	}
	
	// Update is called once per frame
	void Update () {
		
	}

    public void OnTriggerEnter(Collider other) {
        if(other.tag == "Player") {
            Debug.Log("I is here!");
            inRange = true;

        }
    }

    public void OnTriggerExit(Collider other)
    {
        if (other.tag == "Player")
        {

            Debug.Log("I is gone!");
            inRange = false;
           

        }
    }
}

Dolphin Character

using UnityEngine;
using System.Collections;


/// <summary>
/// This code defines the motions and animations of the dolphin 
/// The source original source code comes from the purchased asset
/// link: https://assetstore.unity.com/packages/3d/characters/animals/bottlenose-dolphin-128643
/// </summary>

public class DolphinCharacter : MonoBehaviour
{
    Animator dolphinAnimator;
    Rigidbody dolphinRigid;

    public bool isLived = true;

    public float forwardSpeed = 1f;
    public float turnSpeed = .3f;
    public float upDownSpeed = 0f;
    public float rollSpeed = 0f;


    public float maxForwardSpeed = 1f;
    public float maxTurnSpeed = 100f;
    public float maxUpDownSpeed = 100f;
    public float maxRollSpeed = 100f;



    void Start()
    {
        dolphinAnimator = GetComponent();
        dolphinRigid = GetComponent();

    }

    void FixedUpdate()
    {
        Move();
    }


    public void Hit()
    {
        dolphinAnimator.SetTrigger("Hit");
    }

    public void Bite()
    {
        dolphinAnimator.SetTrigger("Bite");
    }


    public void Death()
    {
        dolphinAnimator.SetTrigger("Death");
        isLived = false;
    }

    public void Rebirth()
    {
        dolphinAnimator.SetTrigger("Rebirth");
        isLived = true;
    }





    public void Move()
    {
        if (isLived)
        {
            //dolphinRigid.velocity = transform.forward * forwardSpeed * maxForwardSpeed + transform.up * upDownSpeed * maxUpDownSpeed;

            //if(forwardSpeed&gt;.1f){
            dolphinRigid.velocity = transform.forward * forwardSpeed * maxForwardSpeed;
            //	if(Mathf.Abs(rollSpeed)&lt;.1f){
            transform.RotateAround(transform.position, -transform.right, upDownSpeed * Time.deltaTime * maxUpDownSpeed);
            transform.RotateAround(transform.position, transform.up, turnSpeed * Time.deltaTime * maxTurnSpeed);
            //	}
            transform.RotateAround(transform.position, transform.forward, rollSpeed * Time.deltaTime * maxRollSpeed);
            //}

            dolphinAnimator.SetFloat(&quot;Forward&quot;, forwardSpeed);
            dolphinAnimator.SetFloat(&quot;Turn&quot;, turnSpeed);
            dolphinAnimator.SetFloat(&quot;UpDown&quot;, upDownSpeed);
            dolphinAnimator.SetFloat(&quot;Roll&quot;, rollSpeed);
        }
    }
}

Dolphin Controller

using UnityEngine;
using System.Collections;
using System.IO.Ports;

///

/// This code specifies how the user input affects the dolphin’s motion
/// The source original source code comes from the purchased asset and includes
/// the first line in the Start() function and the FixedUpdate() function
/// link: https://assetstore.unity.com/packages/3d/characters/animals/bottlenose-dolphin-128643
///

public class DolphinUserController : MonoBehaviour
{
DolphinCharacter dolphinCharacter;
private static string port = “/dev/cu.usbmodem146401”;
private static int baudRate = 9600;
SerialPort sp = new SerialPort(port, baudRate);
public Camera cam2;
private bool cam2Enabled = false;

void Start()
{
dolphinCharacter = GetComponent();
StartCoroutine(“DolphinCircles”);

}

void Update()
{
if (!cam2Enabled && cam2.enabled)
{
cam2Enabled = true;
sp.Open();
sp.ReadTimeout = 1;
StopCoroutine(“DolphinCircles”);

}